[This Transcript is Unedited]
National Center for Health Statistics
3311 Toledo Road
Hyattsville, MD 20782
Agenda Item: Introductions – Review Agenda
DR. CARR: Thank you for being here. I would like to convene the working group on HHS data access and use. What we do is we go around the table and say who you are and where you're from and whether you're a member of the working group. So I'm Justine Carr, Steward Health Care System, and chair of the working group.
DR. SUAREZ: Good afternoon everyone. My name is Walter Suarez, I'm with Kaiser Permanente and I'm a member of the working group.
DR. FRANCIS: I'm Leslie Francis. I'm at the University of Utah in Law and Philosophy and I'm on NCVHS's Privacy Confidentiality and Security subcommittee and I'm a member of the working group.
DR. GIBBONS: I am Chris Gibbons. I'm from Johns Hopkins in Baltimore and I am a part of the working group.
MR. CROWLEY: I'm Kenyon Crowley with the University of Maryland and a member of the working group.
DR. ROSENTHAL: I'm Josh Rosenthal with RowdMap and a member of the working group.
MS. QUEEN: I'm Susan Queen with ASPE, and staff to the working group.
MS. KANAAN: Susan Kanaan, writer for the committee and the working group.
DR. VAUGHAN: Leah Vaughan, member of the working group.
DR. MAYS: Vickie Mays, I'm a member of the full committee and I'm a groupie of the working group.
DR. COHEN: I'm Bruce Cohen, a member of the working group, a member of the full committee from the Massachusetts Department of Public Health and co-chair of the Populations Health subcommittee.
MR. SCANLON: Jim Scanlon, I am with HHS Office of Planning and Evaluation and I'm Executive Staff Director for the full committee.
MS. GREENBERG: I'm Marjorie Greenberg, I'm from the National Center for Health Statistics, CDC, welcome to NCHS, and I'm Executive Secretary to the committee and I'm also a groupie to the working group.
MS. JACKSON: Debbie Jackson, National Center for Health Statistics, committee staff.
MS. SEEGER: Rachel Seeger, Office for Civil Rights.
MS. JOHN PAUL: Tammara John Paul, NCHS, CDC.
MR. SOONTHORNSIMA: Ob Soonthornsima, member of the NCVHS committee.
MS. BEBEE: Suzie Bebee, ASPE.
MS. JONES: Katherine Jones, CDC, NCHS, and committee staff.
DR. CARR: Great to see everyone. We have had the opportunity to integrate the excellent input that we had at the first couple of meetings. And I think that we'll have an opportunity today to really have a working group with some deliverables or suggestions for HHS at the end of today. So what I wanted to do was just go through where we are and who we are.
As you can see we have here listed the membership of the committee, and as you heard Bruce Cohen, cochair of Populations, Leslie, Co-chair of Privacy, Walter co-chair of Standards, and Paul Tang couldn't be with us today but he's co-chair of Quality.
And then what was interesting to me is just the richness of our other members with great expertise in IT. Kenyon Crowley, Bill Davenhall, Josh, and Peter Hudson, not here. Mo couldn't be here, but their tremendous expertise in IT.
And then our other expertise. Wonderful that Chris is here today in his expertise in community health informatics. Patrick Remington has called in but not today I guess, and Kevin just had a baby, and Leah with her experience in social media.
And Chris, you weren't here for the first two meetings but it's been a learning process for these disparate groups to come together and get to a common ground, but I think we've made great progress. Our liaisons of course are Susan Queen, Ed Sondik, Nialls Brennan, Jim, and Marjorie.
So our initial charge is that we monitor and identify issues and opportunities to make recommendations to HHS on improving data access and innovative use, including content, technology, media, and audiences. Second, advise HHS on promoting and facilitating communication to the public about HHS data. And finally, facilitating HHS access to expert opinion and public input regarding policies, procedures, infrastructure, to improve data use. So that's our big charge.
In the details I've just highlighted seven things that we are asked to do, and began with reviewing current portfolio of HHS data, monitoring trends about new information dissemination, social media, identify and monitor types of data and information that's needed by all participants, improve data access, promote and facilitate creative communication, facilitate HHS access to expert opinion and public input, and to advise HHS in understanding and evaluation of how HHS data is being applied and the value it is generating.
So where we are on this ambitious agenda is starting with we had two webinars reviewing some of the data. We're up to half of the data that HHS has. And I mentioned, Katherine, if we could put links on the Working Group SharePoint site, it would be helpful to be able to go back and look at some of these data that we've seen already, and we'll be arranging additional webinars to go over the rest. And actually at the end of our meeting today, 4:30 to 5:00, we have a CDC presentation on some of their data.
But we talked last time about the fact that we wanted to use the opportunity of being face to face to really work and save the presentations for outside this meeting time.
So when we last met we talked about thinking about the two sides of the house, the supply side and the demand side. And so when we think about the supply side what we want to do is take some of the data that we've seen, and as said, identify and study areas of opportunity to improve data access and application. And Mo couldn't be here today, but we talked about let's come away with five tangible recommendations or next steps coming out of today that Jim can take back.
So I asked Josh to take us through from the supply side what does the data look like, what's available out there, what can we do for it to make it better. We'll look at some of the front-end platforms, and then we'd also like to go through some of the issues that Josh raised – taxonomy, hierarchy, and so-on, that we who use the data every day take for granted but not so clear to a developer who looks at this terminology for the first time.
So Josh is going to take us through that and then actually building on what Bill Davenhall also had recommended, that we identify candidate data sets to sort of pilot. But I think actually that's a bit of what we're going to do today. And then Susan, we also want to talk about the special considerations related to survey data availability and usability.
Once we kind of get through this side we'll then begin to think about the demand side. And as we said in the concluding comments at the last meeting what do individuals and communities need to make better decisions and improve health.
So let me stop there and open it up for any comments or suggestions, additions, directions. Are we good with that, Jim?
MR. SCANLON: A little more framing, Justine. And remember now, HHS has really worked with several audiences over the years: the public health community, the research community, the health care community, and that's really where our efforts have been focused.
So it's almost a business to business kind of an arrangement. You need a fair amount of analytic capacity to take the data, use the data, interpret the data, and apply it to the levels.
With this initiative what we're trying to do is – all of you have heard this before – liberate, democratize, all of the data that we have and probably other places, to make it available to a new set of audiences, and using more customer friendly technology, and certainly the latest technology.
So we put together the group – all of you were hand-picked by the way – but we put together because you know sort of both sides of the equation. What we've been doing at HHS for example is besides the usual outlets and ways of disseminating data. The policy now is to put our data holdings into HealthData.Gov - and so does EPA and the other health agencies and the federal government – and to put it there hopefully in a format that developers, applications specialists, and others could take. And really without us having to intervene could just go from there, use the data properly, but basically go from there.
And what we're really asking you to do is help us think of additional ways of getting data to HealthData.Gov, but really besides HealthData.Gov, there are a number of other publically available platforms - we're just beginning to learn about them in HHS - where we can make the data available, where you don't necessarily have to know how to program SAS or SPSS or all the others.
So again, these are different audiences. We're not trying to dumb down the research side or the public health side or anything like that, those dissemination outlets will continue in their full glory. But what we are looking for is applications for communities, consumers, patient groups, community coalitions, so they can take data and really use it to improve health and health care sort of in their own settings.
And you're sort of a unique group because we've kind of put together both sides of the equation, and the first couple meetings I think we've had a translation, everyone getting on the same page in terms of technology and data. But we're actually moving along very nicely and we're already beginning to get some very good ideas and we're really asking you to give us additional thoughts and help us think this through.
We've already looked through the portfolio of data from NCHS, as well as SAMHSA, our mental health substance abuse agency, and from the Agency for Healthcare Research and Quality, which many of you are aware of. So we're about halfway through.
I think at the end of the day, later in the meeting we're going to have CDC talk about its portfolio as well. And then we have a couple more agencies that I think we'd like to brief you on. So then you'll have an idea what it is we have. CMS was one of the first.
And these are all different kinds of data. Some of them are survey data where identity has to be protected, research data, surveillance data. But in CMS's case it's claims data, administrative data, literally from all over the country, probably the most complete set of local data that we have. And clearly that data is being used for hospital compare, to can compare hospitals to other places as well.
But again I think everyone would agree that we are not experts in HHS in the many current and evolving technologies and platforms and really ways of getting the data out, and that's what we're looking to you for.
DR. CARR: Okay. We have a couple of people who just joined the table, if you could introduce yourselves. And then Josh, you want to come up and start your presentation?
MR. QUINN: Matt Quinn from NIST, staff to the committee.
DR. GREEN: Larry Green, member of the NCVHS committee.
Agenda Item: Discussion of best practices and practical suggestions for release of "open" HHS data
DR. ROSENTHAL: Good afternoon. Hi. So three or four days ago I spoke at an industry thing with Brian Seebach and Niall Brennan and some other folks who were on my panel, and we came up with the ultimate device for displaying data securely.
And you're going to see Hopkins professor from MPH and a bunch of other folks, at BCDSCO trying to figure out exactly how to use this, but we put our slides and our analytics on this and this went over like gangbusters. You guys remember the old view masters you used when you were kids? So that's what they're doing. The Hopkins guys trying to figure it out.
But that's not what we're going to be talking about today, although that is funny, that tickles me ever so greatly. At the last HDI thing we had huge foam cowboy hats and you can see all the HHS guys running around in those as well, some industry folks.
so I'm just going to take 15 minutes and be done, in and out very quickly, but I just want to set up where I think we've been and where we're going and what we're doing. So this is our third meeting. The first meeting we got our charge from everyone including Todd, and he was very fiery and inspiring. And then out of that we all sort of left, or at least I left trying to figure out what was going on. And we talked a lot of meta talk about what we could do.
And so I just went together in sort of true startup fashion and slapped a bunch of stuff together. So I put together seven immediate things that I thought would be pretty good in terms of recommendations. They're probably wrong, they're probably backwards, but at least these were seven very specific things. If you walked out of here with nothing else at least we've done that part of the charge.
And then there's various specifics talking about what taxonomy is, network through file, showing how to go about doing that, learning centers and why those are important. Business value, baking that into the challenges. Semi and synthetic data sets, we have an IT person talking about that. Data browsers, this means the dissemination. Partnerships and products - and kind of mocked Todd there as well. Then a green button with an opt-in kind of like the blue button.
So that's what we did, and then Mo and company took it off and said let's actually try to make this reasonable. But this was the first cut as far as that. And so then what happened was over the past few weeks we took a tour. So those were just initial points. And a few of those were around taxonomy, and engaging other people and business contacts as part of the questions, so we weren't just doing cool fluffy stuff. And then also disseminating the data to different sorts of audiences.
And as you remember my background is also in kind of public health data from doing the Dartmouth algorithms and whatever, so I have a firm foot on that side of the house, but also on the startup side with Harvard and MIT and Hopkins, some of the other places doing some of that sort of stuff.
And so we walked through these tours of the data that the government agencies have, and we'd talk about data.gov, and basically there we kind of dumped files and we dumped datasets and maybe we structure it and maybe there's some meta data. And then we took tours of two different things, either a big file or a database like a claims identifiable database that has restricted usage, or kind of a small file that is kind of all over the place and it's really tough to figure out what the pieces are in it, or these kinds of secure enclaves.
And to do that there are kind of supposedly user facing because you can create your own table, but to do that to be honest, you really have to know what you're doing to go in there and most of those were restricted access anyway.
So if the question is how do we engage other sorts of people to use this data? I suggested that there might be a couple things we might want to think about and that's what we're going to talk about for the next ten minutes or so. So a couple things. Does that sound like a fair, accurate summarization?
So I'm going to walk you through some real life resources as best I can, but I'm also going to frame it up. And this file will be available for you as well. And again this is just kind of first draft, putting it out, I'll send that on to you guys. And this is talking about data browsers and really building community exploration and creation of intelligence and market value around information, not data. And so I'll walk you through the examples.
But if we're going to frame up the choice, what can a user – and I'm using that very broadly – do with government data? Users can do a couple different things. If you're very sophisticated and you have specialized expertise and you have access, you can go in and pull the files down from .gov and figure out what they mean, or maybe you're granted restricted access into an enclave when those fire up and you can play around and create your tables, and that's one thing. The amount of users that do that, I tried to poke around, we're talking no more than thousands at the very upside.
The other thing is you can build apps, and that's what data liberation and datapalooza has been about up until this point. We put the data up there and then we allow the app builders to come and do their thing. And theoretically they create market value, just like what happened when we put weather data up and when we put geolocation data up. That really hasn't happened as people will kind of know from looking at CrunchBase and the internal databases that HHS uses to evaluate, in terms of creating market value that hasn't really happened.
Part of that is because they're mostly based by tech people who really don't understand health care, and partially it's because it's DTC, which historically hasn't worked very well, and partially because in order to do that I can't really do that, and I can program pretty well but I have to have a decent amount of money and go into the app store, and maybe I build a team, I mean it's not easy to do.
So I always point to iTriage, and that has about seven million dollars of venture funding behind it, and that's one of the few successes around that. So that's a pretty high bar.
So when Todd talks about putting the data out there and letting the communities build the interfaces that's still a pretty select group of people who are going to do that, not just because of the technical skills which a lot of healthcare folks don't have those technical skills, but also because on the tech side to understand the healthcare data and the nuance you're looking at, those tech folks do not have those skills.
There's a whole informational educational conversation we can have, and that's kind of the last session we did at HDI where we did that sort of information for folks. But that's very difficult to do. The third option is what you can call a data browser, and I'll talk about that, or kind of an interactive front end. And in this scenario the data is already defined, all the meta data is done for you, the taxonomy is done for you. These are wildly popular in the tech world.
In fact one of the more recent contests which was one of the bigger ones was comorbidities of diabetes actually using some of your data and it was wild and no-one really knew about it. And the beauty of that is you play with the information rather than the data. So you don't need to do anything except drag and drop things. So we're going to talk about that.
Just to give you a sense of scale, one of the things I've been trying to figure out is who uses this data and how many people actually use it. And we've tried to get some internal stats from CMS and other folks and that's sort of been processed and what have you. So I am going to roughly, roughly 80/20 proxy it. When I go to Alexa, and this is a good place to go if you want to figure out how popular a site is, just FYI, I know you guys know this but it's a good place to go.
So this is up here, and go to Alexa.com, and this whole slide there will be available to browse at your leisure. So what you see is this is where you type in a site, you basically see how much traffic it has, what its rank is, how many people are linking to it, and you can look at different metrics. Data.gov has about 3000 sites linking into it, and that's a pretty good metric. That tends to be better than hits, but you can use your own stuff.
We're going to look at Google Public, and most estimates of Google Public data, they only say about one to five percent of the people using Google are using Google Public data. But that's still a lot of people. There are five million sites linking into it so you have millions and millions of page views every day, every hour.
MS. QUEEN: May I ask another question? What do you mean by sites linking into it?
MR. ROSENTHAL: So if I link to something, if I have my little blog or CHIDS put out a little post on the economic form you're talking about, and they kind of link, they link to Data.Gov. So they put up on their website or their blog or a tweet or anything on the web, they send a link and say hey there's this cool thing called Data.gov, go here. Or it's anything specific, it's anything that belongs to that domain. It may be check out the hospital compare site inside. So this is just ball parking it.
What I'm trying to say is Data.gov has a couple thousand sites linking to it. Google has millions and millions, and only a fraction of those are using this Google Public thing I'm going to talk about, but my point is that's still a lot of people, that dwarfs the amount of people. I'm just trying to get a sense of scale of what we're talking about.
And then there's these big tech sites, just to give you a sense, when they have these data explorer contests like ReadWriteWeb, they have about 40,000 sites linking in when they do that. And so you're talking more than ten times as many as Data.Gov, and that's just one instance. I'm just trying to get a sense of scale of when we talk about users we try to get some internal metrics and that's still really good to do and to hunt down but I'm just using some external metrics, back of the envelope, right?
This is kind of like the Google Consumer Surveys example I was talking about, you can run a consumer survey using their algorithm for a couple hundred bucks instantly, and it doesn't give you what you'll find out about it but I'm just trying to get a sense of scale.
And so Google Public Data Explorer, let me hop out of here and show you. Actually let me show you the slides before we hop into the live environment. Basically you just search for this thing and you go in and it has these dynamic graphs, this is like the TED talk where you click the play button and it does all the stuff for you, it animates it.
DR. CARR: I am going to take Susan's lead and just ask like the dumb questions to just bring everyone along. This is the browser. So you went to public data, and what's the data that's populating this?
DR. ROSENTHAL: So there's tons of public data and you can search by data types. The World Bank, almost every government institution has had their data taken out and sucked into this thing. Private data, public data, individual data sets, world health organizations, a lot of its international data or it's US data that's not health. There's a ton of census data in here, there's a ton of economic data.
DR. CARR: This is a site you go to and you can pull in the data set that you want, or do they give you here the things you can choose from?
DR. ROSENTHAL: They do this all for you. So if I were to go to this probably the best way to show you is just give me 30 seconds to show you.
DR. COHEN: You said they don't have any health data?
DR. ROSENTHAL: Very little. They have some, I'll show you what they have. But not nearly as much as what you could have. So I go here, I go to Google, I'm going to show you live how to do this.
MR. QUINN: How does Google populate this? Do they have a team of people there that search for this data? Or is it done by computers?
DR. ROSENTHAL: Both. So they have some automated things which Google tends to do, and they scrape a bunch of stuff, and then they have a team of people, and then they have users contributing this. Say if I'm at the World Health Organization, there's someone at the World Health Organization who puts the thing in a CSV file and dumps the stuff. But they have to do three things. Typically they have to give you the data and they have to define it in this taxonomy.
Taxonomy is a fancy way of saying table of contents, and I'll show you what that looks like. And the beautiful part is it's not just a data set. Once you put your data up there I can pull in economic data from World Health, I can pull in US census stuff. Once you put your data up there that mix and mashing, right now if I want to mix and mash health data I as a developer have to take it and I have to go and find census data and I have to go do some Google Survey data and I have to scrape it with some other stuff. Google does it all for you as do all these public data browsers. And so once you put your data up there anyone can mix and match it with anything else that's in there, and the stuff that comes out of that is crazy.
MR. QUINN: How can you make sure that Google finds it or Google gets it?
DR. ROSENTHAL: You'd want to contribute to it. World Health Organization contributes a lot of stuff to people because they want people to use their data and they think it's good to do.
DR. CARR: Just to bring it back to us, if with the HHS stuff some of it is there but there's more that could be there, then we could initiate and push that to that site.
DR. ROSENTHAL: Oh, yes, they would love it.
MS. QUEEN: What was the third thing? You said give the data, define the data.
DR. ROSENTHAL: That's about it. You give them the data, you have to define it in the taxonomy and you kind of have to label it. So they basically ask you to tell people what the data means. So when you say morbidity – here, I'll show you. So I log on here, and here's these different graphs that people have put together. And I'm going to look at this one. I'm going to hit explore the data. And on this computer it will take a long time.
DR. CARR: What you are looking at it looks like it says fertility rates, but that happens to be one data set that you pulled up.
DR. ROSNETHAL: Yes. So here's all the data. So here's the data that's going into this particular view. So this is world development indicators and I can search these data sets in different views and mix them and match them. This is going to take a long time to pull up.
DR. FRANCIS: Is there any way to tag the data so you know the data contributor?
DR. ROSENTHAL: Yes. It's funny you mentioned that.
DR. FRANCIS: So you can tell interrelationships between sources?
DR. ROSENTHAL: Yes. That's exactly what this is over here. So these indicators, that shows the data set contributor. That's different than the person who put the view together. The person who interprets the data and basically says you know what I think – I'll flash over to this one here if the computer fires up. So she won this little contest and she put together this thing. And so what she looked at was diabetes and these things she thought were the greatest comorbidities. And so she pulled essentially from Google Data and other places and tried to figure out what she thought had the greatest correlation, and now she has her name on it as well. And so she kind of became an expert and people tend to look at her and et cetera et cetera.
So there are two levels of transparency. There's one: where's the data coming from, so show me just the stuff from World Health Organization, or from US Census or another source. Or show me by type what am I interested in, or show me by the person who actually pulled the stuff into view.
This is Google doing all the stuff for you. The next layer on this is individuals using it and creating their own stuff. And here I can search by country. So here I'm looking at fertility rate and life expectancy, and I want to subset that by different types of lending types. I might want to pull in what am I interested in. I'm interested in COT emissions to see if that has anything to do with fertility rate. And so on and so forth. I can pull any piece of data in the system.
DR. CARR: Can you show us?
DR. ROSENTHAL: Yes. This is going to fry this thing but let's see.
DR. GIBBONS: What level of data is this? Is this county, country, what level is this?
DR. ROSENTHAL: Yes. This is the beautiful part about this. When you have a taxonomy, and this is actually far more helpful I think actually showing people than me just rattling on like last time. So when you have a taxonomy you're sort of defining your data, and one of the attributes about that is geographic region or unit. and so you have a separate part of the taxonomy, the table of contents, that talks about that. And so you link up country to region to HR to whatever you want to look at, CMS contract, country, et cetera, et cetera.
All of that is built in as a dimensional taxonomy. And so the answer is different data sets have different grains, and if you want to look at the view that only shows you the highest minimal grain, so you can't go down below what you shouldn't see. So before when we talked about how do we do grains, do we want to do zip code, this is how they do it in the non-healthcare world and it works pretty nice.
If you want to submit something you can say I want to submit that, I don't want to do zip plus four, I don't even want to do zip, I don't even want to do county, maybe let's do CMOS contract, maybe we'll do HRR, you can do that. And it works like a charm. Once you figure out that master table of contents then it's very snazzy.
DR. COHEN: You don't hide the data from the public, you just submit the level you are comfortable with sharing?
DR. ROSENTHAL: Yes. That is absolutely well said. And so what you'll find is that this may be a public set. If you wanted to be really snazzy you could do a synthetic set on it. If there's something that you think is very meaningful that you want people to wrestle with but they don't understand the concepts but you're scared of sensitivity or mosaic effect you can use a synthetic set, or you can just use the public sets you already have.
But the beautiful part about this is it doesn't require me downloading an excel file and dabbling around in SAS or whatever I'm going to do. It doesn't require me going on and even trying to figure out how to generate tables on my own. And there's this kind of compounding learning. When I put it up there the next thing I can pull in any of these other sources. And you can see by gender, by urban, by country, I can play the thing. I think what I'd like to show you, let me just take you on a little tour and then we can ask any questions because I want to put all the stuff in your head before we take specific questions.
So here we can look at metrics. Let me look at say health. So I have my data sets I'm saving here, let me say I want to look at world development indicators, maybe I don't want to look at that, maybe I want to look at other elements.
MS. QUEEN: I just wanted to add something to what Bruce was saying. It would be my assumption that something like BLS or Census that have made their data available, they already have determined the lowest level.
DR. FRANCIS: Actually, that goes to my question. If I submitted at this level because I don't want it going any lower, but somebody could mix it with another data set and get it lower.
DR. ROSENTHAL: They are going to typically enforce that maximum. So Google is pretty good at security stuff. I know the government is good too but Google is pretty good. And one of the ways they typically handle this is that maximum grain. You can basically set that in as a rule, as a business engine rule, and say anyone who uses my data set in combination with anything else, the minimum geographic setting is either the maximum of the data I submitted or a separate setting. So here I just searched and I found all these different things, out of pocket expenditure, when we look at to your point the data provider, who do I want to look at? World Economic Forum, Human Development, et cetera.
And here's what they have on health which looks like a lot, but I'm looking right here and this is global obviously and you can click on that. Condom usage, contraceptive prevalence, depth of hunger, diarrhea treatment, HIV, et cetera. The beautiful part is you can mix it with the other stuff in there: economics, lending, social media, et cetera, any of that sort of stuff.
And what would be really interesting is if I'm a researcher and I have a question, hey is condom usage and Twitter usage related on Friday night, I can get an answer to that question pretty quickly without having to write a grant and do that sort of stuff. That's probably not the best example but it gives you a sense of the sort of things you can do.
So that's Google. Google sort of does that stuff for you and you have to work with one of their reps, but they love the Government.
DR. GIBBONS: Can I ask a question? So are we assuming that data integrity is all very good?
DR. ROSENTHAL: They do a pretty good job, and usually they'll rely on – if you think you can do better, if they don't like it, they won't post it. They're pretty quick about taking stuff down if you don't like it.
DR. CARR: Is there a checklist that they have that assesses the data integrity?
DR. ROSENTHAL: They used to – yes, you would probably have to talk to them right now. There are folks that run that sort of thing. And historically there are kind of forms or templates you submit. It's not just a manual checklist, it rejects it if it doesn't do certain things. And then they run things. Not to get too much into the weeds but maybe the best example for the NCHVS people is to think about Google Consumer Surveys.
If you go under Google Consumer Surveys you can ask anyone anything and basically you pay a couple hundred bucks or something stupid like that and at the end of the day they enforce all of your survey stuff using Google's pre-existing algorithms of specificity. So they're leveraging Google's intelligence on top of a survey platform. And that sounds like those things don't go together but they're really good about doing that sort of stuff. And they sort of do the same stuff with the data as well.
Google is really good about data integrity, they're much better than probably anyone you've ever met, so when they pull your stuff up there it's going to be in the best shape it can be. But at least historically as I said if you don't like it they do it in partnership and they take it down. But this is a private entity doing this stuff for their own users, and they're creating these views and this platform. That's very different from the next level which is folks like these.
Here's ReadWriteWeb, and this isn't anything special but it's just one of a dozen highly read technology blogs that talk about data and healthcare is on their pretty predominantly. And so they're not health 2.0 they're kind of bigger web 2.0. And what they said is they'll give someone a pass for free to their version of ATI and $500 to get some tacos and beer if they win this prize.
It wasn't an application challenge where someone had to develop iTriage or something like that. It was just we're going to put some data up there, we're going to label it and put it in a meaningful table of contents i.e. taxonomy, and then we're going to allow people to play with other types of data and see what insights there are. The prize isn't can you build a cool app that does something that people may not use, the prize isn't can you take data at a granular level and come up with a better algorithm, the prize is can you come up with some interesting insights and literally do it in a way that lets other people explore your work and build on it, disagree with it, et cetera, et cetera.
And so they put up a little form and they partnered with these other guys, not Google but Tableau, although these were the guys that the Google founders at Stanford they built this company after they taught Larry and Sergei how do to the search stuff, so they're pretty good. And they basically step one, two, three, four, you submit, and then the person who won – so you can use any of this data that we looked at – so this got a lot of traffic, and what won out of everything, and this was like baseball stats, anything you can think of, crazier stuff than I would dare mention in this company.
And what won was an interpretation meaning a view or a pulling together of data that allows you to explore it of US obesity and comorbidities. And she won, and she was a student at DePaul I believe. And there she is and she talks about it, and her tagline was supersized caskets which is pretty snazzy.
She's interesting because she has no technological skills, she couldn't develop iTriage, nor is she really MP nor could she go through and say hey this is unwarranted variation in this HR and we think that's a comorbidity. And she put together this little thing. And so this is an explorer and she pulled the data and she said here are the things I want to look at.
And so here is the US and color density indicates RAID and so the beautiful part is this little thing went viral, went all the way around the web. I can take her view and I put it on my blog, I put it on my Facebook account, I put her interpretation of analysis in this public data everywhere. They don't have to go to Data.Gov, it floats all around.
DR. CARR: What does this tell us?
DR. ROSENTHAL: She tells us the things she thought were most correlated with obesity and diabetes. And so these are the factors that she likes.
MS. QUEEN: How did she determine which factors were most related?
DR. ROSENTHAL: She went through these and she probably did a regression analysis one by one. So she dragged, just like in Google I showed you the little dot bubbles and did the regression analysis with the flick of a wrist. And so the academics will obviously say I disagree with this or I do this, but here's the interesting thing, here's an 18 year old kid talking about comorbidities in diabetes.
And actually if you look at what she did she actually had some fairly high-falootin folks, some of whom are in government, judge her work as part of the criteria, which is interesting. So she talks about fast food restaurants that would typically take a five year study to do. I mean I know we have food atlas and food deserts and we've played around with that.
So she pulled from that set and flipped it up there real quick. Income, mileage to a store, a fast food restaurant, poverty rate, convenience stores with gas she was using as a proxy for fast food, low income receiving SNAP, price of low-fat milk. So she was looking at stuff that you might not look at typically and she's able to do it.
DR. COHEN: She used existing public data sets at the state level probably?
DR. ROSENTHAL: I think it goes down to county I believe. Yes, county. And in fact there's low obesity, price of sweetened drinks. So whoever asked the question, and I'm assuming, and this is typically how they do this, so she looked at probably 50 things and she said with low obesity price of sweetened drinks, if it's a lot of money for a sweetened drink, consumption of fruit and vegetables – and she's doing this from various sources – adults meeting activity guidelines, meeting household income, full service restaurants, et cetera et cetera.
And the point is not to look at the specific example but just to look at someone doing this sort of stuff. And the beautiful part is she's using different sort of data. She doesn't have to code, she doesn't have to program, she doesn't have to build an app. And she's just interpreting information, some of which is public and is out there and some of which should be out there. And that's her.
And then various folks from kind of the New York Times, they have their new departments who do just this sort of stuff. So it's not just on the fringes of the tech community, even in the mainstream media community this is big. So if I go into the gallery I can look at different things. If I look at health you can browse this on your own. Geography of diabetes, contributors to obesity, health care cost, tooth decay, and I can go on. I encourage you to play with this on your own.
So when we're talking about making our data available and thinking that someone has to build iTriage there's this whole other movement going on. And so very quickly I can filter. Here's what these guys did. This is Annette Griner and she does very good work actually.
So if I were going to hire a consultant or if I wanted to identify who were good at doing something I would look. It's just very much like doing a piece of code in the tech world. I can see how good they are and see what people think about them and review them. And let's filter it by poverty rate and obesity rate and ethnicity and see what happens.
MR. QUINN: Can you identify the underlying data sets?
DR. ROSENTHAL: Yes. So with Google you can and with these guys you have to download the client install rather than just the web-based one, and so you can absolutely do that. So what I tried to show you was two different approaches to this. Not to say that you have to do it one way, I tried to show you the two ends of the spectrum. Other people do this too. You can think of it as YouTube or Kickstarter for data and analysis. So Google takes an approach where they say everything has to fit in my table of contents if you give it to me.
And the benefit of that is they don't create and define the views, they allow you to pull anything else you want to at the drop of a hat. Tableau takes a different approach which is ironic considering their shared Google heritage. They basically say it's scattered. The data that you give us is a freestanding little unit with referential integrity only between the sets that you submit as a block.
And so if I were to look at this geography of diabetes versus tooth decay or versus healthcare cost I can't link back and forth between these sets. And so that means that the individual who sets it up has more flexibility in terms of the metadata structure in what they want to do. The downside means someone else can't come along and talk about baseball stores, do they have a correlation, which believe it or not there is some interesting stuff in there.
MR. QUINN: I was going to ask, Google publishes its taxonomy, correct?
DR. ROSENTHAL: Yes.
MR. QUINN: I didn't see any nations, I mean Google being worldwide, have any nations or any public health entities in the world said we are going to shift our vital and health statistics or other national statistics to accommodate Google's taxonomy?
DR. ROSENTHAL: I don't know the answer to that. Historically, their approach has been to do it collaboratively. So what they say is if a subject matter expert or a producer of the data knows something and feels very strongly about doing it in one way they typically accommodate that or don't show it.
MR. QUINN: In the world of building standards, you start with something that everybody can agree with you hope, and adoption results in standardization.
DR. ROSENTHAL: Back to my slides, I said kind of taxonomy, and we said what's taxonomy, it's really whacky, how do we do it, et cetera. And I said well actually people have your data in a taxonomy right now. You might disagree with it, but there's a starting point, there's a rough draft you can start from.
And what I'm trying to show you is there's different ways of putting this data out there that has different types of adoption. So this user is not a hardcore tech person. And by the way, as you do your own public health initiatives you should really look up there, there's some really interesting stuff believe it or not. Nor are they necessarily a data analysis person.
So in terms of kind of doing contests and incentivizing people I used that little ReadWriteWeb, I'm only using it as an illustrative example. Here they're giving away $500 and a free pass to their own conference so it basically costs them very little, and they partnered with a technology platform who was all too happy to partner with them. So outside of having a dedicated person in-house do it they weren't spending a lot with that and they got massive traffic per Alexis and some of the another analysis.
So what I was trying and hope to show you was you can do this sort of stuff with very little investment. I mean the users or the partnerships do it for you. And it reaches a very different sort of audience. And you can do that in two ways. You can say I start supply or demand side with the data I feel safest with, put it out there and see what happens, or you can say actually we're really interested in this subject and maybe we do a synthetic file or something we feel secure with at a particular grain and see what happens, what conclusions do they have in a particular topic of interest.
And what you can really do since there's only five or six kind of secure access passes granted to different enclaves you can say whoever wins the contest using that synthetic file judged by HHS - and the people in this room if you want to do it, so you can have your hand in seeing if it's worth doing or not - actually gets access to the real data. So there are all sorts of fantastic creative things you can do. So that's it.
DR. FRANCIS: On that slide, there was a place to click download, and what I'm interested in is what can be downloaded, and if data are downloaded is there a way to continue to enforce the levels thing when it is downloaded.
DR. ROSENTHAL: What you can do is if you are working in the Tableau world or people like them you can do whatever you want to do. You can basically say we don't want this downloaded, or you can say we're not comfortable with it downloaded, do something synthetic. Or you can say the only grain we're going to do is whatever you want to do. And they typically download extracts.
So a good example of this is run a free Google Consumer Survey and just see what happens, because the download you'll get – and this is a good analog or actually look at this – is a PDF of insights, so they do all the correlation for you. A survey I run rather frequently is what people think about Medicare Advantage. And I do it by open users, I do it by enrollees, and I do it by current users of Medicare advantage. We do that for our private thing. And it does it all for me.
And so in this state women with this education and this income actually have – it does it all for me. And I can download the PDF of those informatics or it gives me just Excel sheet basically, and their grain recently went from state to county.
MR. SCANLON: Rather than peer-review processes, this relies on the community of users to judge the quality or the accuracy of usefulness, right? And open peer review I hope process.
DR. ROSENTHAL: Yes. And they do it in different ways. So Google has kind of – not surprisingly, this is why they get criticized for kind of being like Microsoft. So they're much less peer-review based than say Tableau. So they actually are much less open. They actually, because they're a closed system, their experts internally review it. But Tableau takes a different view and they basically open it up.
DR. CARR: This really is remarkable for how disruptive it is for our traditional ways. And I think about the meeting that we just had over the last day and a half, we think a lot about privacy, that was a lot of our discussion, and it's interesting that in some ways they've solved it. Like we're not going to allow you to get to too small of a level. So it's not debatable, it's just done.
And we're not going to allow you to merge. And I think the thing that's interesting is this is hypothesis generation in a way more than a peer review article. And I think it speaks to how we learn in this new environment. We talk a lot about how many journal articles there are versus 10 or 20 years ago. Now we say how much information is available and minute to minute it's not possible to make a world of peer review articles over 18 months or whatever, 20 years, how long it takes to publish an article.
So this is extremely informative as we think about the data and now where it goes and how it's used. I mean the examples you've showed, the diabetes one is just powerful. We've had lots of article about growing obesity, but when you see that and you juxtapose with all of the community factors, again it goes back to for NCVHS because we have our diagram of the influence of not just the person and the community and et cetera, this is the embodiment of that kind of diagram.
DR. ROSENTHAL: They don't have your expertise and they would obviously value that very highly. The other thing I should say is that hypothesis generation or education. So rather than how are you going to field the knowledge workers and et cetera, you can get a traditional MPH and that's interesting, or you can go to Corsair edX, MIT and Harvard put their courses online, Stanford has a data mining certificate for like $10,000 you can get a certificate from Stanford and you're certified as being able to mine data and you play with some of the stuff.
And when I've been hiring in our product development I've always taken the latter candidate rather than the former. So the point of the story is there's a whole bunch of different things in here that are worth considering.
So I hope this is helpful because the first time I was talking about stuff it wasn't terribly helpful because this was the mental model I had. So feel free to dismiss it or change it or do whatever, but I just wanted to give a sense of the type of stuff which is out there, and I think that can be a real credit or something that the committee can contribute. Because right now health data initiative is about putting data out there, letting people use enclaves, and helping the app developers do it.
And then you know what happens? The guy building the where's my parking app also just takes the hospital compare, that doesn't really solve a business need. By business need I mean also public health, public social need, reasonable value, however you're going to do it, where this is kind of a different way of going about it.
DR. COHEN: This is great. Thank you for letting us see how your mind works. I see this really as evolutionary rather than revolutionary. From the public health perspective first we generated huge stacks of reports of numbers that sat on a shelf. There was probably something that preceded it, probably just filing statistics and books, and then we generated the reports when we learned how to bind and when Xerox invented photocopying.
And then a lot of folks at a variety or perspectives – my perspective is government – have produced web-based query systems at the state level, at the local level, and at the national level. And the focus there was nobody reads the report so let's put them online somehow through these web-based query systems, and that's been perking along for about 15 or 20 years.
Some of the state WDQSs are more sophisticated than others, and I see this as the next version in that progression of disseminating information. This is just, to me, an easier to use easier to access web-based query system that gives the user a lot more flexibility because the contents, the direction are predetermined by the data holders.
So we're saying we've got all this neat information, let's get it out there, some in more combined formats and some – why don't we put all the vital statistics at the county level from the US, give it to Google, and put it up there and let people work on it?
Or if we wanted to focus on a project that's more directed, why don't we take heart disease mortality and behavioral risk data on heart disease risk factors and what we know about the social determinants related to heart disease and work with somebody you think and put a data set out there and see what folks do with it.
DR. ROSENTHAL: That was sort of when we were doing the webinars. It was kind of asking the questions, are they thinking about this sort of stuff in these ways, it didn't go very well so I tried to back off. That's exactly what I was trying to articulate, much better said.
DR. FRANCIS: I want to go back to the point Justine made about privacy. From that privacy perspective the interesting questions are what level, if I'm a data contributor, what are the levels I want to put in?
And another sort of version of that is is there a way that I can make sure that there isn't a proxy for something lower than the level that I'm comfortable with those results when you aggregate data sets. And I simply don't know the answer to that, but you're shaking your head, maybe there isn't any way to do it. But I'm assuming that there must be, so you don't get – I don't know, I'll call it GIS coordinates replacing zip codes or something like that.
DR. COHEN: We started with the basic building block. Let's say county, which most people accept, people of New England really relate to. Then you can essentially build robust data sets and where the data are too sparse there are a variety of ways, and you can have some blank counties or there are techniques to impute county level values. If you get aggregate county level values from a variety of indicators depending on the cross-county classifications, we can throw as many different data sets on top of those for people to look at simultaneously without increasing the chance of identifying any one individual. So I think looking at aggregate data overlaid is a different strategy than linking in individual level data that increases the probability.
DR. ROSENTHAL: That question came up in a previous meeting, and so in a slide, not this one I gave you but a previous one, you'll see privacy comp side, personnel side, health care, I have her quote up there, a professor on that specific thing at one of the conferences I spoke at, and so she's addressing that and essentially saying the same thing in much more complex language.
And number two, here's a crazy idea: You say we don't have that download button. It's literally a closed environment. The only reason you need to download data is to be able to pull it down, link it with other things. But if you can do it all in a browser you don't need to download it. So those views I was showing you, one of them had download and one of them didn't. That's what I was trying to say earlier, if you can do it all in the web you don't need to download it.
DR. FRANCIS: That is where there are going to be really hard questions about what you allow there, too. For example the SAMSA example that's in the minutes raises exactly that question.
DR. ROSENTHAL: That is why I started with the view master, that was sort of tongue in cheek.
MR. SCANLON: This is not for everybody. Obviously the research community, the public health community, they're going to continue to get the data they want for their own work and then run analyses, hopefully in better ways. But this is another attempt to reach a whole new set of people, it's not so much business to business, it's more business to person.
DR. ROSENTHAL: Some of the stuff planetRE(?) does is a version of this, and Tableau, they do a lot of B to B. So Fair Isaac, your FICO score comes from using this. So this is the public or kind of dumb version of the thing that actually supports the very high end B to B stuff right now.
MR. QUINN: The thing that really struck me about this, and this has got my brain going, is this taxonomy. That is the key to all of this and anybody who's ever said, hey, let's develop a taxonomy and get everybody to use it, you can just here the whooshing sound of time, and it's 20 years later and you're no closer. And this is the key to putting all the data together.
DR. ROSENTHAL: And notice their taxonomy isn't like a metadata of taxonomy, it's data, it's the business taxonomy. In your original files you'll see an example of a business taxonomy of our data. What is this, how do I use it to answer business or performance questions.
MR. QUINN: It is more normalized.
DR. ROSENTHAL: Yes. So it's not just this is the ICD-9 code, this is answering specific questions.
MR. QUINN: You can always build it out for that. That's an area where I would love to see communities in specific areas – for example public health, or specific areas where there is the need for metadata or semantic detail.
DR. ROSENTHAL: The community kind of has three approaches to it. Typically the academics is to say let's get in a room, let's forma standards committee, we know what's best, and that's going to be our taxonomy. And that tends, historically, not to be the most efficient way to do it. And there's also some questions about the integrity of that as well.
The other option is just to do a purely open source community which is what Google and these guys have done already. The other option is build a community of subject matter experts. Every MPH, grad, et cetera, with peer reviewed et cetera. You have different options.
MS. QUEEN: Josh, on your previous slide I think it had the contract number. This was where at the last meeting when Ed Sondik was here, there was a definite disconnect for those of us in the survey world who don't even know who in the agency or where in the agency you have the information that's needed. You knew what you were talking about, I don't think we understood. Where do you get that kind of information?
DR. COHEN: I think the best solution is to take a data set, take NHANES or take mortality data or take a Medicare claims file and sit down.
MS. QUEEN: But NHANES is a data dictionary. What you're talking about is something different when you're talking about contract number.
DR. ROSENTHAL: This is payer compare data with Medicare and MA contract.
MS. QUEEN: The surveys all have the detailed variables.
DR. COHEN: It is not going to be rocket science to create this taxonomy, the file formats and structures exist, they're just not arranged in a way that developers think about putting data into applications.
DR. ROSENTHAL: The beautiful part about this is once the developer does that then – this is what I was trying to say at the other meeting, if I want to do this right now I have to as a developer do it myself, and then the next developer, and then the next developer. In this scenario, I do it once and then I put it in the browser, and then everyone is living off the fruits of my whatever. You can have an official standard or you can have multiple people using different things.
MS. QUEEN: But there is one big difference with the surveys, and I know we'll talk about this later, the survey data has to be run to generate. So you have to run it, you have to weight it, when you download these data files you're not going to pull out a record from that, you're going to run it to get whatever it is that you're looking at. If it's health insurance you may have six different health insurance questions.
DR. ROSENTHAL: You are essentially giving them an analytic extract, and in that Google Public data there is results of kind of international survey data in there.
DR. CARR: If we were going to then – because we have our charge of coming up with specific recommendations – what do we want to take away from what we see here today?
DR. ROSENTHAL: Can I say one more thing and then I'll be quiet for the rest of it?
DR. CARR: Don't be quiet, that would be foolish.
DR. ROSENTHAL: Don't tempt me. What I was trying to show with this was originally in the recommendations we talked about kind of taxonomy and learning center, all this stuff, and the question was what is the unit around the learning. Is it around the data? What's the definition of contract versus HR? No, that's not so helpful. Is it around this, that, or the other thing? Originally when I made the screen cap in your other slides of doing kind of HHS and CMS data driven product development around seeing what people use, this was a piece of a broader learning center.
So I didn't originally envision this as its own thing, free-standing, but to your point whoever asked about authorship and community, you have this functioning as a layer in that. Someone working with the data and saying how do you do it, someone working with information and saying how do you do it, and then someone on top of it saying how do we actually build applications for public/social good, business value, et cetera. So I just wanted to throw that out there, I don't want to overly narrowly focus on this. This was just kind of a piece of the puzzle.
DR. COHEN: I understand the context for what you're doing, but we have to start somewhere. And to the extent that starting small, the first goal is can our charges deliberate the data. How do you start liberating the data? You start by getting it out there and see how it resonates and then working up from there, I think it will create its own context rather than trying to think about the grand scheme and work down.
So in terms of a first step these tools exist and they're well used by everybody apparently except folks in the public health field. And representing the feds here, we're not familiar with these, we haven't used the value of these in getting our data out for people to do the things they want to do with it. So in terms of a first step I think this is a wonderful strategy.
MR. QUINN: The thing that strikes me with this is how about we choose a data set that we're comfortable sharing with Google, with the world, and see if it fits the taxonomy and work through the process of putting it out there, see what happens.
MS. QUEEN: I think with the Google one, because I was playing with it, aside from Census I think something from AHRQ is already up.
DR. ROSENTHAL: They pulled it.
DR. CARR: They pulled it meaning they took it down?
DR. ROSENTHAL: They scraped it.
DR. CARR: I need taxonomy. What does scraping mean?
DR. ROSENTHAL: They have these little automated things and they kind of crawl through the web and they pull information down in structured ways and then they can post the data.
DR. CARR: It is data that is already on the web and they pull it into their application?
DR. ROSENTHAL: Yes.
MS. QUEEN: It was something from AHRQ. It was either maps or HCAP or whatever's been made public. So with HCAP you don't have a public file.
DR. COHEN: Wearing my community hat, the problem is with surveys because of level of granularity. I would choose a more surveillance type data set like births or deaths or cancer where you can populate data from every county.
MS. QUEEN: It may have been the HCAP.
DR. COHEN: I think that would resonate and reach a broader audience.
DR. ROSENTHAL: I would only say, just while we're thinking about it, is that ReadWriteWeb example is very interesting. You have people at HHS who are pretty good at business liaising, who know some of the people at places like ReadWriteWeb. If you're going to do it you have a lot of cache, don't kind of post it on your own.
Depending on how big you want to go, if you want to have a million people using this data set if you do the PR and marketing around it – in the first meeting we talked about kind of PR and marketing around it, I just wanted to throw that up there as a lever. You can post it up there kind of in the night and then it'll be up there and no-one will really know about it, or you can go big and say the winner of this gets the HDI thing, et cetera.
DR. CARR: Wait. I didn't really understand what you said. Let me break it down. We can put data up on the web, we could put it on Google or we could put it on that other site, and we'd be better than we are today because more people would have access to it. And then you're saying we could draw attention to it as part of a prize or something like that?
DR. ROSENTHAL: Yes. If you approach in a public/private thing, and there's people at HHS who do this. There are challenges, and it's worth thinking about the marketing, do you want to do it stealthy and kind of put it up quietly or do you want to make it --
DR. CARR: Well probably all of the above. But do the one thing. I mean if we could make it available to these sites, Google and what's the other one called? Tableau. We could do that, and that would be light-years ahead. That would open that data up to orders of magnitude more people, so that would be huge. Then we could be better than that by drawing attention to it.
DR. ROSENTHAL: Relatively easily for 2013 there's the application challenges, those are on-going, they're planning it right now.
DR. CARR: You are talking about a Datapalooza?
DR. ROSENTHAL: Yes, as we speak. And the challenge is going to be who develops the best application. You already have your data up there. My point is there is very simple things you can do to broaden the users. Not just people who want to use the data and introduce challenges to the applications.
DR. CARR: They are not exclusive, though, putting it up there is number one and then drawing attention to it is number two.
MR. CROWLEY: Part of the value in this too is being able to mix and match different data types. So rather than thinking about one data file now they want to think about a couple of different data files that are being used in different ways, and then see what works how and what doesn't work, perhaps different opportunities within this data structure.
MR. QUINN: I just think the process of going through the process of selecting a data set or two or three or four or whatever, and actually walking through the steps of doing it, posting it and seeing how it works and the reaction, just the government process of doing that, and then the process of seeing if people find it, the feedback that we get, that's valuable in itself.
MS. QUEEN: I am wondering if with the Google site for example, if for those data that have already been made available, public use data files or public data sets that are already up there on HealthData.Gov or Data.Gov, Google presumable could go and download them as part of their site, is that correct?
DR. ROSENTHAL: Verify everything I am saying because things have changed in the while since I looked at this.
MS. QUEEN: If they are publicly available they've already been through all the proper disclosures so it can be downloaded.
DR. CARR: I am going to be the person asking the really simple questions, but I think that's a great point. So if we have already made this data available on the web, then Google could go and take what's there, right?
DR. ROSENTHAL: Yes. This will get into a business discussion really quickly, so you're going to have to have someone representing the agency talking to someone at Google representing it.
DR. CARR: We don't need to get to that, but just conceptually.
DR. ROSENTHAL: Conceptually if you're going to do it, or if you're going to spend a dollar, you could hire a third party to do the scraping and formatting for you.
DR. CARR: In other words HHS will be part of the process of that data going onto that website, whether it's proactive or reactive?
DR. COHEN: All the mortality data at the county level by age, race, sex, and cause are already up online via WONDER, but WONDER is pretty much a cult product because it's difficult to use and people don't know where it is and how to see it.
MS. QUEEN: Isn't it a tool? That's not the raw data, correct?
DR. COHEN: The raw data exists back into the system because it's got all the death information at the county level.
MS. QUEEN: The distinction I am making is from the viewpoint of somebody who is trying to download something. Like when I query BRFSS, I do the online tool. I don't download the data, I just get the results of my query.
DR. COHEN: That is right. I was responding to the notion that there are data that are publically available via these tools, so the underlying data exists there and I'd say they're all on parole, they haven't been liberated.
DR. ROSENTHAL: In terms of feasibility, of what's possible or not, I think that's a pretty good place to kind of get a subject matter expert to come in and talk about it, or I'd talk to Google directly.
But if the question is do we want to pick a set and try to do something with it, before you go into that I would at least do a feasibility analysis and say is it actually easier, can they do that without us doing it already? Do we want to do it to maintain control in certain ways? Or instead of doing that should we take an alternative path like some of the private scrapers I was talking about?
DR. CARR: Let's try to map out a roadmap here. We've seen about a dozen data sets, 10 or so, that are available only in these structured sites of HHS. So now we would want to move some of them, maybe all of them, to a Google data platform. So one is to do that we have to have the right people in the room from HHS, and the second is that they would define the parameters of protection, et cetera, sample size.
If we made a recommendation back to Jim that the data leads in each of the agencies and in HHS do exactly that, that they meet up with Google and report back, which ones are a go, and if any are holding back, if so, why? I think that would be a very valuable next step for them to do. I like the idea of piloting the data but I don't see a need to pilot anything.
If that's all out there then let's move that forward. If we thought about a pilot, I think then we would be in kind of a different domain of either responding to here's a need and this data could be put together in a particular way. Or maybe, as you said, cardiovascular or an app or a business case, or something like that.
So they're kind of two things. One if simply continue, complete the liberation by making it more digitally usable.
MS. GREENBERG: In this brave new world I'm this old-school executive secretary of this committee. Not of this working group, but of the committee.
So I guess we talked a little bit before lunch about what are we talking about, and I still, from the point of view of how this working group would convey information, I still think if you're talking about recommendations that have to go through the national committee on vital and health statistics, if you want to make some suggestions like we heard this presentation, this sounds kind of interesting, we might want to explore this or something, that's fine.
But I really, just being sort of steeped in all the policies, et cetera, of several advisory committees, I really don't think that this working group can make recommendations on its own to the department.
DR. CARR: Sorry. I am sensitive to that. As a reactor panel we could perhaps have the reaction that this front-end browser could add value to the datasets that are already out there.
MS. GREENBERG: That might be worth exploring. I'm from a really data for dummies point of view, so I'm more of a data policy person than a data user person. But if you take data that are already on the web, people can download them, maybe it's not so easy, but the intent is not to make it difficult, the intent is to make it available. I recognize that it may not be that user friendly to the uninitiated.
So if it's already there I'm trying to think what's the downside of putting it – let's just start with the Google rather than the Tableau – but what's the downside? Well, one is that you asked the questions, what do you have to do?
I mean a question of can Google just come and take it, it sounds like they couldn't because they would need this taxonomy. They're not in the position to write out a taxonomy, are they? I mean they have a taxonomy, but how do they put these data that they don't know anything about into this taxonomy?
MS. QUEEN: Marjorie, for all of the data that are downloadable from HHS that are publically available, along with there are links to the information, so the information is there. To me the difference is if Google has done this with data we've made available they've done it with data we've made available.
For HHS to intentionally ask them to do it or sanction it or whatever, it's sort of like HHS would have to be approving what Google does. There is a distinction between having it out there and letting them use it and encouraging their use if anybody uses it.
DR. ROSENTHAL: I will just say three things. They can absolutely do that right now if they have the intention, and in fact they've already done it and they do it with a bunch of different bodies and there's very little you can do about it unless you want to get very feisty with them.
MS. GREENBERG: Why aren't they doing it, is it too much work?
DR. ROSENTHAL: They have done it, with some data sets. There's more to be done for sure. That's why I say there's almost like a business context to think about. They can do this on their own, do you want to encourage them to do that as you redesign Data.Gov and do the next generation of it?
Do you want to just encourage them to do that? Do you want to have some sort of working partnership with them where there's some sort of intelligence or security you're putting on top of that? Or do you want to kind of do it yourself and take the more active role?
That's kind of the decision you probably want to think about, or one way to frame it up. They don't need your help to do that, they can do that on their own quite well.
MS. QUEEN: Isn't that what NIH is doing with Amazon? Something with the genome data, the big data?
DR. ROSENTHAL: Yes. That is just one way to think about it. You can pretend you never saw this and just kind of let them go on their own, and they'll put more up and five years from now you'll look and a bunch of your stuff will be there, or you can say we want to actively encourage them to do this, we want to promote it and we want to get users to do that, and that means some very specific 101 marketing and PR stuff that we're thinking about. Or you could think about becoming even more actively involved in terms of saying we have research and public health agendas, we want to put certain things forward in terms of et cetera.
MS. GREENBERG: I'm asking those of you like Susan, who are more familiar with Data.Gov, we have high confidence that even though by putting it there or it being there, we must, because they can do it now. No matter what other data they mash it up with or whatever, it just isn't going to be a problem.
MS. QUEEN: Well, we have all of these privacy and compliance requirements that the agencies go through, a lot of them are very similar but some of them there may be a suppression of three to a cell or five to a cell, it may vary depending on the agencies. But they have a lot of different statistical disclosure techniques that they apply to the data before they are released.
And that was one of the documents that I put up on SharePoint was something that we had to develop as we put together for OMB when HHS took over HealthData.Gov from GSA. So it described all the different techniques that the agencies use to protect the data and to minimize disclosure of anyone's identification. So that's out there.
DR. COHEN: I agree with Josh. I mean, what's the business case? The business case is we want to proactively liberate the data to expand its use and to make it easier for folks who haven't traditionally used our data to be able to get to it because they use these tools, they know these places, they don't know the indicator warehouse.
All the data at the community level in our first presentation in the indicator warehouse, all those data have already been reviewed and populated pretty much mainly at the county level. There's a huge amount of information out there.
If we push that into Google as another platform for release, or if we want to create patterns using Tableau, combining those data with other data that traditionally don't fall in the public health sector, I think that's the goal of data liberation, really.
So it's not protecting our enclaves, and there's already a lot of information that is quasi-public, but people just don't know how to get to it because it's not in a place where they go to look for information.
MS. GREENBERG: What is the downside?
DR. COHEN: My answer is there is no downside for the agencies or for the community, and particularly I think this will encourage agencies to think more creatively about how to disseminate information in their data.
MS. GREENBERG: Is the only reason we haven't done it is because we don't know about it, or we don't have the resources?
DR. COHEN: It is not a priority. We spend 99 percent of our time making sure the data we collect are of high quality. We spend one half of one percent of our time thinking about dissemination and getting it to the people that need it. And that's just what we initially thought – I'm speaking now as a government person – what our charge and responsibilities are.
We fulfill our obligations by being the stewards and making sure the data is collected and protected, and once we're comfortable with that we need to flip the switch and send our children away.
DR. CARR: I just want to make a comment. I think Josh said it in the beginning, you have a finite amount of time to spend and you can spend it trying to figure out how to manage the data, or you can outsource that to Google and now spend your time thinking about the content and the value.
DR. GREEN: I want to join the Susan and Justine line of dumb questions. In this arrangement, what is a data analyst?
DR. ROSENTHAL: I don't do very well on philosophy so I have difficulty answering what strikes me as metaphysical questions.
DR. CARR: Where are you going with that Larry? Maybe elaborate a little bit more.
DR. GREEN: It is not a philosophical question, it's a job description. What's the job description of an analyst once you get here? In the old world we'd whine and complain about public health departments having very little analytic capacity. And we paid premium bucks to someone who can develop mastery of a data set, understand it in depth and breadth, understand what can be linked and what cannot be linked, et cetera. So in this world I'm just asking what is an analyst going to be.
DR. CARR: I think if we get back to the life cycle of data or the supply chain of data, I think we have moved the nidus of control from one venue to another. And I'm not sure that an 18 year old can tell us about diabetes. So that's already happening whether we planned it, liked it, or whatever.
But I think we have to think about then what is after that hypothesis generation, what happens to that hypothesis. It looked pretty right to me, it looked good and there's a lot of stuff behind it, but someone else could juxtapose a lot of data that was not as sophisticated and that too will be out in the public domain.
DR. ROSENTHAL: In this sort of interactive business intelligence world outside of health care and even in health care, not necessarily at MHP world, typically there's kind of business analysts, there's business intelligence analysts, there's metadata, they'll called a metrician.
So one way to think about this is it's become more specialized. The person that sets up that taxonomy has arguably a much higher degree of skill. If I wanted to do that five years ago I had to run thousands of lines of SAS. That's all I did, I sat there and did that. I was an analysis but I was running SAS all day, I had to run a regression.
The person who sets that up is largely doing the same thing, they're just not doing it by hand. There's a greater degree of specialization. There's not so much a generic analyst but there's difference types of analyst nowadays is the easiest way to describe that.
And in terms of the fundamental tasks are very similar. One way to think about it is the speed and rate has increased. So if I'm running an analytics group which I have, and one of the analysts comes to me and I don't like his factorial analysis basically, I think his correlation is off, I don't think he's correctly done a feature construct set and come up with reasonable comorbidities for diabetes, I'm essentially doing the same thing that took me three months to do in two days now, but the functions are the same.
DR. CARR: I think Larry's point is that you have a group of people under your control, and you supervise them. But unsupervised people have access to these data as well. And that's a reality, and that is the disruptive piece around this.
DR. COHEN: Perhaps disruptive, but essentially we're moving the data upstream to put it in the hands of the decision makers rather than having to create this artificial interface between the people responsible for decisions and the data, who we used to call data analysts. I mean that's the goal of data liberation.
Decision makers make decisions whether they have data or not. Our underlying assumption is by providing more data directly to decision makers they'll be able to make better decisions. It might not all be decisions that we agree with, it might not all be decisions that we would have made given the data.
But our goal is to provide decision makers, whether it's individuals choosing health plans or whether it's community groups deciding what priorities to operate on, to give them the opportunity to use the information that we've generated to make decisions.
So who is the analyst? The analyst now becomes the decision maker, and the information is not filtered through any lens other than the lens that they want to use to make decisions.
DR. ROSENTHAL: I apologize. I misunderstood the question, I thought we were asking what in terms of the job description is an analyst working for an analytics shop. There you have a group working.
But in terms of the open sourcing, absolutely. The specialization is you don't need to require everyone to understand the taxonomy. One person can do that and then you can open it up and everyone else can build on that rather than hierarchical. And that tends to work pretty well, at least that's the charge as I understand it.
DR. CARR: I think your point is excellent that it gets into the hands of the decision makers. So if you get information, let's say by your community, and you say this doesn't resonate with my reality, you're going to be motivated to enhance, refine, or improve that data set so that you haven't missed anything. So it's kind of crowdsourcing or something, it's a lot of people weighing in on something.
DR. FRANCIS: What I want to actually push on, which I just don't understand, is in a way what's all the hullaballoo about, because if the only question on the table is you've got a public use data set and it's on the HHS website or it's over on the Google website, hey, I mean with the Google taxonomy, no problem. There are no more privacy issues raised than there were when it was on the HHS website.
Now, if I go to what Bruce said a minute ago, more data, and ask what that means without knowing it, or I look at something like the webinar summary from the 23rd where there's some tools to allow public access to do analytics behind a firewall, but you don't get the data because of confidentiality restrictions. Now if what we were doing was essentially taking that data and putting it on Google with the firewall allowing the analytics --
MS. QUEEN: I can never see the restricted data being made available publicly. The restricted access even has a lot of strange things. Certain SAS codes you can't run, there are little proc lists, there are a number of things they won't even let you do, and you can't download it.
DR. CARR: At the continuum of data there is a tremendous amount of data that as Leslie said, is hugely valuable, especially when juxtaposed with other available data that accounts for decades of learning. And they can still be the people who have the restricted access and expertise, et cetera, to use that other data.
DR. ROSENTHAL: Not to cloud the issue, and that's one thing for James to think about, but when we were taking those surveys of websites and what they were doing, if you were in the private world and looking at say credit card data, instead of a portal that allowed you to create a table very clunkily you'd have a professional version of one of these browsers up and running and be able to do that very quickly and very meaningfully and all the researchers could ask questions and get answers and do things intuitively.
So that's a separate subject for how do you get people out there but it is worth thinking about, like when I walk in with my glasses and see this is the generation of stuff we've spent a lot more money on in the commercial world with cost to put one of these things up behind a firewall for researchers like yourself to do, that's just something else to think about, but that's eminently feasible and being done really easily.
DR. CARR: We're at 4:30, CDC is going to do a presentation. Do we have 45 minutes now to kind of land the plane a little bit on what we'd like to put forward for consideration? As we as the reactor panel seeing this information, I just want to make sure we have the right language, Marjorie, that we've fulfilled the charge, that we're not making recommendations and we're not sending a letter, we are reacting to information that has been presented to us.
MS. GREENBERG: I am seeing a frown on Leah's face. Leah hasn't said anything, do you mind if I call on her? Your reaction to this discussion.
DR. VAUGHAN: I guess I am trying to understand what's most helpful for me to help you. One of the things that does kind of strike me, one of the things that might be helpful is to actually get you guys to drive data sets on a number of the platforms outside of your area of expertise and see what you understand and then have to feed it back to somebody whose area of expertise it was perhaps so that you have a sense of a range of tools which are far beyond Google which has its own challenges and problems around privacy, but so that you actually get a sense of what it is that can be done.
And there have been some wonderful examples in the non-profit sector of doing that, there have been some wonderful instances within the challenges of taking specific data sets and putting them forward, including a million parts. But to maybe not just have you drive it but have you drive some data, not scraping it, but drive it, and to just have you have the direct personal experience of going through it.
DR. CARR: I am hearing you say two things. One is that we shouldn't just focus on the two browsers that we saw today and we ought to broaden that to say we ought to leverage public browsers.
DR. VAUGHAN: There is that piece because at private companies there's a very strong open access movement both here and in the UK which is very much about not limiting it to only open access, but to ensure the integrity and continued to the extent we understand perpetual availability of the data sets to the public without them becoming proprietary. And there's been some excellent work done in both, there was just a conference last week. But my strongest impression is that rather than talk about it and make a recommendation we need to actually do it.
DR. CARR: Again, we are a reactor group, and the doing – I mean we're doing a bit of travelling through this today but I think that HHS has configured in a way now that there are individuals accountable for the utility of their data.
DR. ROSENTHAL: What are some other public data browsers that you're thinking of, and could you clarify the difference between driving and scraping for me just so I understand what you're talking about?
DR. VAUGHAN: There is a very large number of browsing products that are available, and I think my suggestion had to do with data domain experts using some of those visualization tools that are available right now to see what data looks like and feels like by using them for this public facing analysis. Well maybe to try it in your own area, but to try it outside of your area so you're perhaps maybe having more of the consumer experience.
DR. CARR: With regard to that part of your suggestion, you were saying for us to have an experience of what it was like – I mean Josh was like let's pull down this, let's pull down that –-
DR. VAUGHAN: That is not the same as you doing it.
DR. CARR: You doing it yourself?
DR. VAUGHNAN: Yes.
DR. CARR: I mean actually I did it during that last webinar, and so at the end of that experience each person will see what was hard, what was easy, what was unexpected. And then what? Are you saying that would influence which browser?
DR. VAUGHAN: I think – I am hearing some misunderstandings and I think the best way to, rather than just talk about it, is to just do it. I think the notion of doing outside of particular area of expertise and then reading that to someone in that domain whose expertise it is is, again, a little bit more of the consumer experience even though everybody here is an exquisite expert, to see what you're learning and understanding outside of your particular day-in and day-out.
DR. CARR: But again that information would then inform the direction of which browser.
DR. VAUGHAN: Not so much which browser but the ways in which they're useful, the ways in which they're limited, the ways in which there are still privacy issues, some real ones. It's not so much that the data sets aren't hugely refined to ensure privacy, it's that given that it's aggregated to a large area, there's still people who will find a specific address to have that meeting even though that's not statistically appropriate or policy appropriate, it still will happen, and it does happen.
And to a certain extent there is nothing you can do about that, but you should understand that that does happen, and to understand what some of those consequences might be. Vaccine myths are certainly a huge example of that. No matter of truth telling seems to quite absolve them.
DR. CARR: We have a lot of conversation at the meeting about the accountability across the food-chain or the supply chain or the lifecycle or whatever you want to call it of data. But I think it is true that we are now in a world where data is out there and responsible people use it and irresponsible people use it.
And so we talked about the stewardship that when you might be showing data that's alarming, disturbing, what impact, housing values and so on, and you have a responsibility not to suppress the data but to deliver it in a way that is constructive. But I'm not sure that that's where this group is. The fact that that could happen, we've already made the data public and I think the question that we're trying to address is how do we make it more usable.
And with that we're all on the same page that leveraging a browser might not be any of these but exploring whatever browsers are out there to enable a user to focus on the content, not the configuration, would create greater opportunity to learn from the data. So are you disagreeing with that?
DR. VAUGHAN: Not in general, I'm saying that I think in terms of strengths and limitations of many of these initiatives that I think you would come away with a better idea, better able to make those recommendations with an experience of a number of them, and just actually hands-on doing it.
DR. CARR: I agree with that. But I'm not sure that it's the role of this group that meets a couple of times a year to do that homework to advise HHS. I think that highlighting that the data can be manipulated more readily, not manipulated in a bad way, but moved around more readily if you had a browser. And to go to that point to then leave it to HHS whose day job is to make this data available.
DR. VAUGHAN: I think a lot of those things have already happened. I wish Patrick Remington was here today, I don't know if you've had the great pleasure of using his site, County Health Rankings, which I think does an exquisite job of doing just that already. Certainly American Academy of Family Physicians has been a leader in all of this.
I think there are a number of awesome really finely done examples of that already, I think there's great instances in some of the challenges already. Actually I did not read your cross-agencies, but to put forward specific data sets.
MS. QUEEN: Actually, I was thinking that one thing we probably should do, whether it's staff – I would want to look at what challenges that have been done that are ongoing that they're using to adjust data.
MR. QUINN: We could have ONC or someone at HHS do that.
DR. VAUGHAN: It's not just ONC. It's everybody.
MS. QUEEN: The other thing is I didn't have enough time and I was just playing around with the Google site for example trying to figure out what Census data, what BLS data, I was kind of in a hurry and I could see AHrq, so getting a better sense of what's already being used on these other platforms, I mean I would want to know, it's just something I want to know.
DR. VAUGHAN: In terms of the international data, the world bank is really just done an amazing turnabout in terms of how they open up their data and how they make it accessible, I certainly commend their initiative to your consideration.
MS. QUEEN: I think the HHS Chief Technology Officer in his office would be involved at least with all the challenges at the Datapalooza. I mean we have a source at least for the things we know about related to challenges and this other stuff, we could be looking ourselves at least some extent just to find out what's already being used.
DR. VAUGHAN: As to the specifics of Google, it's a multi-tiered process directly involved in the Hurricane Sandy initiative right now and how that data and that set are handled in other parts of the company. So there are many people I'm sure who would be delighted to talk with you at Google, but it's a fairly complex process and a large company and hard to generalize about any single thing.
MS. GREENBERG: I sort of have mixed feelings because I don't want to be a stick in the mud at all, and I want to encourage us to be thinking creatively and not spending a year studying things. At the same time I do think that even this working group, although it's formulated or established somewhat differently than the National Committee, does need to have sort of a deliberative process in which you gather information.
And if not doing the work yourself because I have to kind of agree with Justine that I'm not sure that that's – I wouldn't stop anybody playing around with it or whatever but I don't think that's solely the work of this group – so that if you just say well we heard about this, we think it'd be interesting, we really have to question how far that would go, I mean what kind of influence that would have.
On the other hand if you sort of write up this discussion, chop it around the entire working group because we don't have half of them here I think, maybe we'll be able to stop hearing about data sets but actually hear about some of these different activities that people are doing, kind of enrich the discussion, find out about challenges, some of these other things.
And then taking it – you can stop there – but taking it to the national committee and having a discussion there could result in something that people would actually listen to as opposed to – and I know what Jim was saying, the department might want to just bring someone to this group and say, what do you think. Fine, that's a reaction. But if you're the initiators I still say it gets very close to recommendations.
And if it isn't recommendations, if it's really in a low level, oh we just heard about this and we think it might be interesting, I don't know if it would have much impact. So I'm thinking both about whether we've been deliberative enough or understand enough to say something meaningful and then also how it would be the audience.
As Bill said, you have to think about the audience, how is it going to be received. I know that you want to come out with something and not just be talking heads, but I really am wondering what really – I'm trying to think about what would be most useful.
And I said if it's at a really informal just kind of suggestion level which is really all that I think would be appropriate given that you are part of a federal advisory committee, that it should not be making recommendations on its own without going to the full committee, then it seems to me that it might not be overly useful.
But you can try it and see I guess. We've heard about these activities that obviously have potential, they're of interest, we hear there are other activities, there are people who might be able to tell us more, some of the pitfalls, some of the challenges, some of the issues. I think it may be that there needs to be a little bit more deliberation frankly.
DR. CARR: I do want to get your feedback but I want to hear from the Working Group members. Why don't we start with Josh?
DR. ROSENTHAL: I am not so savvy in all of this, I'm just working from the charge and I just want to know what the charge is. I thought the charge was to actually make these specific suggestions. So if we're just hear to kind of review, then that's very different than what I had thought.
MS. GREENBERG: No. I didn't say that. What I was saying was there are two ways that this group can be used. One is that the Department can bring you some things and just ask for reactions. You can do that, you don't need to ask the National Committee on Vital and Health Statistics to approve your reactions, your reactions are your reactions, and your input, et cetera.
And the Department has an interest in using the group that way. And to learn really from you because you have expertise that the people in the department and even in this advisory committee don't necessarily have.
But if you're going to make recommendations, then just like any sub-committee, the sub-committee eon privacy and confidentiality just spent six months putting together recommendations related to a stewardship framework for community health data.
They had hearings, it was very deliberative, but they can't just make those recommendations to the department, they have to go through the full committee, it has to be deliberated through the full committee. So I am saying if this working group wants to make recommendations, it also has to do it through the full committee. That's the rules, that's the guidelines, otherwise you're basically violating FACA.
DR. CARR: We have reactions, we heard a number of things, and one reaction is that the data is hard to use. But I don't even want to speak for that. What I'd like to do is go around the room and ask each person to provide a reaction just from what we've seen so far, and then also what else should we be thinking of.
And we've heard about a couple of things today. We've talked a lot about big picture. But I think what Bruce was saying and I think actually what Leah was saying as well was actually getting to what is the demand side, what's a big population issue, what do we want to put together, maybe a data set that then users could use to develop. That's how I kind of think about it.
But let's start over. Josh, a reaction, and then to Marjorie's point, if we want to go in greater depth we can have other presentations, hearings, webinars, whatever else. So did you want to offer something, a reaction, and a next step?
DR. ROSENTHAL: Yes. I'd like to know more. The first set of slides I made available, this set of slides I will make available, I think it would be very helpful if we're talking about things at specific incidences of them kind of brought to the group. So I heard several things in your comments and I have to admit I was pretty confused. If there's a number of different public browsers, I want to make sure we're talking about the same thing because we're using terms.
DR. CARR: That will be for the discussion afterwards. What I want to do is just get a hearing around the room. I think you guys are seeing things from different perspectives and I get that.
DR. ROSENTHAL: I heard that security in a browser is a different risk than actually having the AHRQ file up on the website in its own browser, its own instance, and I don't agree with that from a technical concept. And I want to make sure that A) that was what was claimed and B) that's what we want to really dig into.
DR. CARR: Let's hold that for a discussion. Kenyon, thoughts, reactions on what we saw so far on the HHS data and thoughts about a next step?
MR. CROWLEY: Broadly, in terms of HHS data, I have been quite pleased with the different innovative programs. I mean I think the webinar on the behavioral health data was quite instructive. I mean typically that being sort of a data source that's so hard to get to and so hard to use, having found solutions then I think it's great – and the other sites as well.
But if you generate a lot of questions too, and I think some of the questions to think about, so as that new behavioral resource is being put forward and other resources that are made available through the health indicators warehouse and through HealthData.Gov, and as these are growing, I started to have some questions, including what can I answer with this data, how can I make use of it, how can I share the data with colleagues, where can I go to make more sense of the data.
And I started thinking about that and interestingly enough when I looked through the HCUP data they actually had a full set of these are the full questions you can answer with the data, these are the things you can do, so obviously there have been a lot of smart people thinking a lot about these issues.
Now I think in terms of the charge of looking sort of how can we make this more broadly available and create this learning environment, so I think the committee should continue to think sort of closely about how can we make sure that the learning data is captured as people are using the new data source and using existing data sources is made available to others, in that sense.
So maybe something as people are looking for data that if they do find the data they need, if they don't find the data they need, how do we know whether they do or they don't? And if they don't what mechanisms are in place to allow to point them to to allow everyone who is very familiar with the data, maybe within HHS, or more broadly within the community, to say this is your data research question, this is what you were trying to do, I know how to do this. I've used this, this is one way to do that.
But creating an infrastructure using sort of social media or sort of other architectures that have been used for other open source communities, that allows those questions to be moderated by a community and to have that learning and those answers within the community being fed back in to the community so that as others come to use the data they can more readily create the most value from it. So those were some things that struck me.
And as a reaction to the discussion today, I think that data browsers provide an additional channel for people to more readily use data. As it was discussed earlier, it's not for everybody but for many people who are visual learners and maybe not be experts in SAS or other techniques, having the ability to mix and match data and view those results somewhat instantaneously can – back to the analyst question – can sort of accelerate the analytical abilities of a much larger population base.
So essentially you're taking what was a public health analyst or other type of analyst with a very specific set of skills and training which is still very valuable but you're using technology and decision systems to enable those same types of results and findings to be accomplished by a different set and a wider set of people. So I think that's important, something that should continue to be looked at closely.
MR. QUINN: To bring this back – so the charge and the focus of the broader NCVHS effort is around the community as a learning system, using local help to improve local data. So this report talks about how we're missing. We heard from a variety of communities that are doing various things to improve health and use data in their communities, but by and large we're not building the infrastructure, the technical infrastructure but also the data analysis infrastructure in communities that's needed to improve health.
And the broader NCVHS is looking at solutions for that, strategies and recommendations for HHs to address that. I see this as directly involved with that, and ways of reducing the burden on local folks who don't have millions of dollars to spend, who don't have the time and effort and resources to reinvent the wheel.
And in this context I think what we could do is we can both understand what's going on today, so are any of these folks who we've talked to or others using resources like this or those provided by the government to address this? Would they like to, is it in the cards?
But also to understand what needs, what infrastructure could be provided centrally or on a distributed basis like this to address those needs and to inform the broader NCVHS on that to incorporate what we learn here through those recommendations for some of our other activities.
Is there, for example, not through the lens of subcommittees but there are other standards issues for combining data or making it available. Are there privacy issues, are there quality issues, to view it through that lens as well and to say our goal is to build the infrastructure to support communities as the learning system, can this get us closer to that or is it an unrelated thing and we should look at it as something else?
DR. GIBBONS: I am still getting my feet wet here trying to understand fully what's going on. But just on the little that I've heard today, sort of thinking about it, it strikes me that there are sort of three issues, and Matt touched on a couple of them.
One of them is the issue that the data is hard to use, and I guess we've been talking about that a little bit. The other is in terms of the same sort of thing, and I don't know the answer to this, perhaps you do. What is the sort of general knowledge in the country about the availability of the data sets just in general?
Because if that's widespread then it's a non-issue, but if it's very small that's different than the data is hard to use, and therefore the solution to that is education or something else. I'm just thinking about what types of problem might be preventing widespread use of this data. One could be that people just don't know the breadth of data that is available, whether it's hard to use or not.
And the last one, maybe it's getting a little bit on the demand side, but before even seeing this and hearing Matt my question was how are you defining community. Are you meaning some geographic area? It has many definitions.
But really what I'm getting at is what it is that we understand the community capacity to use this data, and do communities, however you define them, really have whatever level of capacity and desire? Because on the one sense we may be assuming that if we get the data to a certain level, then wow, 300 million people in this country are going to use it.
But maybe not, maybe it's only going to be a small subset of people who would ever have the interest, capacity, and desire to use this. I'm not saying that's bad or good but I'm saying that may be a reality. And if we have a better understanding of what's the universe of potential people really who would use this level of data then we could target or think more appropriately in terms of what is good and what is best, and where our efforts need to go in the future.
DR. FRANCIS: As I am trying to get my hands around this, as I understand it there are at least three different questions and I'm sure there are many more but there are three that I will outline that I think HHS is facing as it releases data.
One question is what data. Another question is in what format. And a third question is to whom. And some of what we've been talking about right now is in what format, and some of it is to whom, and for example when Susan answered my earlier question about the SAMSA data it was we're not changing the data that we release we're just changing the vehicle and maybe who gets it. But I'm not sure about that.
So what I see the role of this working group as is helping HHS to understand the questions, the options, the benefits, and risks of different options so that HHS makes as informed a decision as it can about what's a really important question.
I'm as deeply committed to having usable data out there I think as anyone out there in the room, but what I want to be really sure about is, and obviously from my questions and how I introduced myself, I'm not a tech person, but I want to feel comfortable that there isn't a reason why somebody who read in the headlines HHS gives its data to Google, I can say to them there aren't any risks here, that's a real misunderstanding.
But I think the role of this subcommittee or working group is to make sure that we first, at HHS, I'll go back to what Leah said, aren't acting under misconceptions because if we are something could happen that would be really scary and that's the worst case scenario that I don't think anybody wants.
MS. QUEEN: Can I just say one clarifying thing because I didn't meant to give the impression that it's always the same. The public-use data files are definitely not the same files as the restricted access files.
DR. SUAREZ: I was trying to get my arms around this, as well. I came up actually with a list of six. Some of them are overlapping, but let me go through those very quickly. I think the very first question is availability of data, what is the data available out there. The other one is a question about reliability, validity, and completeness of the data. Another dimension really is the limitations of the data, that some intrinsic limitations in terms of the characteristics of the data and then some external limitations like privacy related policy constraints.
So limitation to data is the third. Barriers to access, I think that's a very significant one. That might be technical, that might be policy. And then the last two are really sort of more once there is data, once people have data, it's really the tools to improve usability of the data and then the ability to aggregate that data. And the last item is really a mechanism to improve the analytical capabilities, sort of the data analytics and the resources that exist to this data analysis.
Now these organizations like ours, Kaiser's and others, are dealing with this big question of the big data issue and the whole data analytics. And I think the same type of challenges are going to apply to communities that are going to be trying to use in some way big data.
In this case virtualized perhaps but still dealing with the same challenges of the three V's as they are called, the velocity, the variety, and the volume of data. So those were my areas I guess that I think there could be some additional work that this workgroup should do.
MS. GREENBERG: I have already spoken, but I'm really concerned Josh. I don't want to sound like I'm putting a wet blanket at all on what you said, because I think we've all learned some things today, and I encourage that. I'm just trying to think through how you can be most helpful not only to the Department, but really more broadly to the public and to communities who, as we all said, need to use data and want to use data. I think a lot of the issues have been raised. So I think that's all I'll say right now.
DR. COHEN: Where to begin? I actually like Leslie's framework for what it is that we need to do. So the question for me is what are the next steps? Essentially the most specific question is what are the options for leasing HHS data at the county level to maximize its utility, to make it as visible as possible, and to let it breathe and have as much value as we can.
And the three questions that I think Leslie raised are, what are the data we're talking about releasing. We began a review but essentially that's pretty easy to answer, we can go dataset by dataset to decide what data and what level.
The second question is how, and I heard some suggestions from Josh today about channels for release, and I think Leah brought up some good points. We need to review other possible channels that exist because we don't want to reinvent the wheel and there may be other ways to get the data out there.
So I think the next step for us is to review what other mechanisms exist that are actively liberating the data. And the third question is to whom, and my answer is to as many people as possible and as many venues as possible and as many ways as possible as long as we're comfortable about the parameters of that release.
I think Chris raised a good question: what is community? I mean traditionally we've focused on geographic communities but certainly we can form the data to target other communities, whether they're race/ethnicity communities or gender or age specific or whatever affinity groups are, it's always possible to re-aggregate and reorganize the data in a comfortable way where we can address a variety of definitions of community.
So I guess next steps are identifying, I think we're moving along the path of identifying what data we're talking about, we need to really review what channels and options exist for releasing the data, and then I think the next steps would be I think really kicking the wheels, trying it out to see how it works, and then making suggestions or recommendations or providing advice to the agencies for what we think would be the best way to liberate their data. And it might require us as the national committee to make formal recommendations to the secretary about how to move this process along.
MS. GREENBERG: Let me just ask, when you said the goal is releasing county level data, are you talking about data beyond that which is currently being released?
DR. COHEN: I just wanted to have a specific target. The lowest level of aggregation that's viable at this point in time for most of the HHS data holdings that aren't address-specific like where homeless shelters are or where halfway houses are is at the county level. There might be other configurations like hospital market areas that people use, it could be MSAs, there are a variety of others, but basically the geographic configuration is the county level.
Some of the data are, as public use data, available at the county level. Other data, vitals for instance, is not available at the county level as public use data, it would require review to release that data. But I think that's possible. So, again, we would need to go data set by data set to see what data can be released. But that's a technicality, that's not an impediment.
MS. GREENBERG: It was my understanding that what this discussion was supposed to be about today was given that the Department has already made the decision to release a lot of data through HealthData.Gov, is there a way to make it more accessible and usable? Not even going to that next step which is also I think within the scope of this group to say are there data that the department has decided not to release or hasn't yet released that maybe could be or how could it be.
But let's just take the stuff that's already out there, and then there are some of the questions that you've all raised. Is it there but it's difficult to use? Are there tools that could make it easier to use? Are there approaches that would allow people to do more things with it?
I have a feeling that there is a very limited number of people who are going to be, even though there are a lot compared to how many used to, I mean we go out from NCHS, we go to universities and everything and say, oh you collect that data? I mean in schools of public health. So never assume anything I would say.
So that's a different question but I think we need to figure out, or you need to figure out, which questions you are asking. And so I would start with the data that the Department has already decided to release and take it from there.
DR. COHEN: We can discuss that in more detail later. My response is the indicators warehouse, HealthDAta.Gov, are good channels and they reach a target audience. It makes more sense to me to bring the data to where we know people are rather than to try to move people to where we put the data, and that's essentially -
MS. GREENBERG: That is what Josh was talking about.
DR. COHEN: That is exactly it. We need to maximize the visibility of the Department's data release efforts, but there are existing channels that people use to seek information and those are the ones that we should be promoting and providing the data through.
DR. CARR: Great.
DR. VAUGHAN: I think there have been a lot of really great comments. I love the idea of trying to understand who the community is and does it make a difference to them or how can we help it make a difference to them and make it more easy.
And to really expand the user-base of the public use data sets, let alone give more texture and bring a more finer grain of data. I think that certainly making it easier to visualize the data is one way, and there's lots of good choices in that.
But I also think that we miss a lot of opportunities honestly without going to the data analysts that exist and asking them if you could, if you weren't constrained, how would you want people to use your data. And I think we don't ask that question and there's a lot of wisdom there and a lot of really great ideas. So to understand that we have a lot more great ideas out there, that we should also be trying to use.
DR. GREEN: I want to make two sets of comments, one is about process and one is about the work. The process thing reminds me of earlier discussions in the last 36 hours. We have such uneven experiences with the technology that's now being used with already liberated data, and I really heard you saying one thing that could help us just work together in a better process if we had a shared understanding of how this goes, and then you proposed a tactic for doing that.
And this connected back to something you said, Justine, earlier in the last couple of days about the groups still in search of a common language so you can just talk to each other. This keeps surfacing in the process stuff. It looks to me like you're making good progress, but I understand what you meant better now than I used to, and it's still an issue.
My comments about the group go like this. It's going to be an echo of Matt and Chris. Where this anchors in the work of NCVHS is how can information technologies help communities to be learning health systems. I mean I think that's still the overarching question that runs through, that's the river running through it all. And I would remind you that that report that Susan wrote, one of the key things is we have a missing infrastructure, and what we're observing is that there are new infrastructures that are emergent, that are known to a few but not known to many, that they're sort of nascent and they're unclear.
We're noticing that often missing standards, they're sort of making up the standards as they go along, we're not sure whether they're good standards, are they useful standards, or they enhance the product, what's the deal? We just don't understand that much.
But you can see how it cuts across the groups of NCVHS pretty readily. Another thing, this stewardship framework, we do not have data stewards. And your conversation just called it out in spades. Who's the data steward for Hartford Connecticut? Particularly given the liberated data and the different people who are using it who we don't know who they are or what they're doing with it or what they're going to mesh it up with and that sort of stuff.
And this is where I'm probably just going to stake out a little personal territory. We're missing a workforce for this. The workforce we've got has a job on their hands to transition to the workforce we need. and that looks like very fertile territory to me in terms of advising the department.
I want to end with two metaphors that you might not find very useful. A lot of the work that I saw going on here today calls out the difference between knowing the map and knowing the terrain. You can map the terrain, and you can know everything about the map, and you can tell someone to go up here and turn right on road 13013, but if you know the terrain, you say when you get to this farmhouse that has a mailbox on it that looks like it should have been torn down 35 years ago, don't go any further. That's knowing the terrain. And we're further on in making the maps than we are at understanding the terrain.
Another one goes back to being a doctor again. My life as a doctor since the internet was invented has changed. People walk in all the time with data. They walk in with maps. They walk in with comparisons. They walk in with tables. You know what they don't have a clue about? They have no idea what they mean, particularly to them. To the particular community what these data means requires contextualization and local knowledge.
So if this group can help understand what the technology can and can't do, the full committee I think, is looking to help move that forward into stewardship frameworks of what's needed to make it work, what can be done to enable it so that the people aren't just reading the map but they're actually making a different in the terrain by doing it.
MS. QUEEN: I have a whole bunch of conflicting thoughts, but the first one is all of the agencies have health data leads rather than a health data initiative. So there is a source of like a listing of what's currently out there that has been made available from HHS.
So I've just been going to the HHS website and trying to figure it out from there. And the health data initiative also has an indication of the granularity of the publically available release data in terms of geographic, whether it's county – most of them aren't county – but there is an available listing that we could use to give us an idea of what's already out there.
I personally am going to be compelled to go to a couple of the things that we've looked at today to get more information on what is being used there and then also just checking with our chief technology officer regarding some of the current challenges or the ones that have already passed but what's been done so far.
DR. CARR: Our discovery continues. I think that everything that was said today, this is a time of enormous change with an asynchrony of available data information and skills of knowing how to use it. And I think that if it feels like it's hard to come up with a simple answer it's because of the enormity of it.
MS. QUEEN: I think we may also want to hear from NIH about their public/private partnership on their big data project since that is an initiative that was announced earlier this year, the Whitehouse initiative with big data. So they've managed to put their stuff in the cloud, we should find out.
DR. CARR: I guess it is harder than we thought to have five next steps, deliverables. But I think we still have a little bit of confusion I think, with regard to what are the boundaries of reaction versus recommendation, and it's just that it's new territory. So I think all that's been said today, and I hope that we can actually get the transcript, or Susan Kanaan is taking some notes, and get it out and try to frame it.
There are multiple directions here. Do we want to choose one direction, kind of go a little deeper on that and come out with something, while planning for the next direction? Clearly the intersection of the work of the full committee on communities and the opportunity that is in front of us here with the data liberation, is important to marry up and we can perhaps take this to the Executive Subcommittee. I know that we have some data, are we going to get a presentation at 4:30 about the CDC data? Is someone on the other line?
MR. BUELLER: Hello. Hi, this is Jim Bueller from CDC, I just joined the call a few minutes ago.
DR. CARR: Thank you for joining us, Jim. We have mapped out about a half hour, does that work for you? 4:30 to 5:00 to take us through it?
MR. BUELLER: Sure.
DR. CARR: Are there slides?
MR. BUELLER: No.
DR. CARR: The floor is yours.
MR. BUELLER: Thank you. I'm Jim Bueller, I direct the public health surveillance informatics program at the CDC, and just briefly we run several of the large surveillance systems that are associated with the CDC, that notify the disease system which is based on reporting that occurs within states, and then states volunteer to share that with the CDC according to conditions that are by law reportable in states, and then the states agree on which subsets of those they'll all share with us. Those data are updated weekly.
We run the BioSense system, BioSense 2.0 which is a large syndromic system which keeps track of patterns of disease they're seeing in largely hospital emergency departments on a daily basis, and we run the behavioral risk factors surveillance system, which is a telephone survey that all states conduct, to track trends in a variety of health risk behaviors or other health care use behaviors. And again, we aggregate that at a national level, and the survey is conducted by states, we support them in doing that.
In addition we provide a variety of services and informatics services, that supports infrastructure of the public health surveillance. But broadly beyond that we are a place at CDC for addressing cross-cutting issues. So while we run three surveillance systems that's a small fraction of the over 100 or so different surveillance systems that are managed by CDC programs, and the vast majority of those are run by different programs.
In essence, if you think of your patient as a population within your jurisdiction, whether that is local or state or national level, the surveillance simply is what we do to keep track of the health of our patient or that population. And the way that that gets done are varied as the spectrum of things that are of concern to public health, from traditional infectious diseases to injuries, to chronic diseases to maternal and child health, occupational health, et cetera.
Whatever the types of issues that CDC programs are addressing there typically some form of surveillance to go with it, which means that they are run by experts in particular diseases or conditions, as part of individual programs and in gear to meet those needs.
And I thought I would just give an example of several different types of surveillance activities to give you a sense of the flavor of that. So for example with the notifiable disease surveillance, as I mentioned all states require that doctors or laboratories and others, report certain diseases. Most of those are infectious diseases, although they also use that authority for reporting to things like cancer registries or birth defects registries or maternal mortality. But for the national system, it's focused on infectious diseases. The states get together and agree which ones should be nationally notifiable, and then they agree to share that data with CDC. They come in weekly.
But in addition to that there are a variety of other systems that complement that. So for example a system called PulseNet, for certain infectious diseases the states may require or they give clinical laboratories the option of submitting isolates of certain bacterial infections to the state public health laboratories.
State public health laboratories then do a DNA fingerprint or electrophoresis that characterize in a very specific level the particular strains and then those data are reviewed at a state level and also shared with CDC. And that provides a way of finding disease outbreaks that are associated with products, typically food products, that are sold in multiple states that might be resulting in disease that at a given state or locality level, would never, at least initially, be recognized as enough of a change to be out of the ordinary but when you pull that data together from multiple states and look at the very highly specific DNA fingerprint, you start to see that something is unusual. So that's how a number of the fairly high profile multi-state outbreaks have been detected in recent years.
Another complementary system to that is something called the emerging infections program, which is an even more detailed project that working in about 10 different states where different jurisdictions either at a local or state level, are funded to do a very comprehensive effort to find all cases of certain infections within a specific geographic boundary.
Trained abstractors go in to perform a detailed record abstraction and get information much more detailed than would be possible through routine reports. Information on the antibiotics sensitivity, information on a specific anatomic site, information on other clinical aspects of the illness that would be beyond the scope of what's routinely collected.
And I could give a few other examples, but the point there is that for a given condition there may actually be a mosaic approach, one approach offering on a broad national level that provides for routine data, and then a supplemental approach that goes beyond that perhaps on a subset of cases, based on whether laboratory specimens are available or based on a specific project that's funded in a few localities to dig more deeply.
And I think you'll see that if you look across the number of diseases, that different surveillance systems operate at different levels. For example another example might be tracking the impact of influenza. Influenza is not necessarily a reportable condition, and yet states keep track of, during flu seasons, the number of visits for influenza like illness.
It's a relatively nonspecific definition that might capture people with some other diseases but that's for the practical need of tracking when has the flu season started, is it worse or more severe than other years. and then death attributed to influenza and pneumonia are tracked at the other level.
At the other level, efforts are made to collect specimens, not at a comprehensive level but at enough of a level to get a sense of whether circulating food strains line up with a vaccine in a given year, whether they're sensitive or resistant to different antiviral drugs, et cetera. You see a similar approach in the chronic disease arena where there may be a need for data that is updated less frequently, even on an annual basis, maybe decision to keep track from one -- to another.
And more and more you see that many systems don't involve the primary collection of data but draw on data that are collected by others. So for example diabetes surveillance tracks do diagnose of different outcomes of diabetes. And they draw on a number of the NCHS surveys or information systems, vital records, national health surveys, NHANES, et cetera, to do their mosaic.
I've been talking so far about information that is generated or arises because people have sought health care in one form or another. Another approach is to do survey. And as I mentioned, many programs at CDC draw on some of the NCHS surveys.
There are also some surveys that are sponsored directly by the other parts of CDC. The major risk(?)surveillance system for example, has been running for about 25 years, we fund the state to conduct it, each state conducts the survey themselves, using a standardized set of questions, and then they have the option of adding any of a number of different modules that could be employed to look at different issues. And they have the option of adding some questions of their own individual state interests as well.
But it differs from NCHS surveys which are drawn upon a national sampling frame. These surveys are conducted by each state, and many times they actually are able to look at sub state levels and they are then used extensively by states to manage their chronic disease prevention activities.
So there are lots of different approaches to surveillance, lots of different systems. They operate on a variety of timeframes, at different levels of detail, different levels of geographic coverage. You can collect a little bit of information about a lot of people or you can collect a lot of information about a few people, and you have a trade-off. But you want it to be timely or complete, whether you have a lot of money to spend or just a little money to spend, et cetera.
But perhaps I can just stop there and see if there are questions and make sure that I'm giving you the perspective that you're looking for.
DR. CARR: Thank you. Susan?
MS. QUEEN: Hi. This is Susan Queen. I was just wondering, are any of the surveillance data made available to the public.
MR. BUELLER: Yes, it varies. There are a variety of considerations. So for example, with the BRFSS, you can go (teleconference operator interruption). Some programs at CDC have the resources to invest in preparation of the public access database. It varies in terms of what's available. Part of the process of surveillance is providing information back to people so they're all producing reports in a variety of formats and ways, but they vary in terms of whether they are public access databases.
And within the public access databases we have to be mindful of what the level of detail that's provided to minimize the likelihood that an individual patient could be identified.
There's also we have to be respectful of whatever concerns the state may have and when they may prefer when people go to them and then to us and information about a particular state. There are some instances where there may be data use agreements within a state between a health department and a hospital that provides data. It is just a variety of considerations that go into that.
DR. COHEN: Does CDC, itself, provide de-identified individual level data for research or public use?
MR. BUELLER: In some instances, yes.
DR. COHEN: Would the IOI surveillance data be available?
MR. BUELLER: That is an example where we don't even get it at the individual level. We get from individual providers the percentage of patients that they're seeing that have IOI. It varies from program to program what level of access or availability they have.
I think one of the issues is it's really what are resources that it takes to prepare and document a public access database. That is going to vary from program to program. There are a fair amount that is available but certainly not all of it.
I mean there are also instances when a public access database may be insufficient for a particular researcher, and there are precedents of researchers working directly with an individual program within various agreements of what would or wouldn't be done with the data. I think you can appreciate the importance of sensitivities about the confidentiality that surrounds many, not all, systems.
DR. COHEN: Is there a summary by surveillance set of what's available and what variables and at what geographic level?
MR. BUELLER: We actually have an inventory of surveillance activities at CDC, and right now that's an internal resource but we're working on making it more broadly available. It's most immediately available also to people within state and local health departments, but we are working to make that available in surveillance activity.
And that does include information about what the URL on the CDC webpage is to go and get more information about that. But it's going to be highly variable from one system to another.
PARTICIPANT: Just one quick question. Just curious Jim, what are your most popular, most used data sets?
MR. BUELLER: That is a good question. We run something called WONDER which is Wide-ranging Online Data for Epidemiologic Research. There's a fair amount of NCVHS data there, there's any number of different systems from CDC, you can access BRFSS data there.
I think some of the notifiable disease data, there's census data there that you can use. I would venture a bet that probably the BRFSS is one of the most heavily used. If you ever want to dig into it just go to CDC.Gov/BFRSS and there's a tremendous amount of information that you can get. I guess BFRSS is one of them.
I know that AIDS has maintained a public access database. I know that from having worked in AIDS in the past. I don't work there now, but I would presume that that's maintained. But there was a lot of interest in that for a number of years.
Obviously the NCHS systems are very, very heavily used and I think it's fair to say the NCHS would say that they don't operate surveillance systems, but their data are used for surveillance and for many other purposes as well. They're really geared and built up to provide public access databases, so they're very heavily used.
DR. FRANCIS: I have only seen proposals for surveillance systems that are operated on a distributed query basis. I just wanted to ask whether the sort of standard model is that you collect the data or whether there are considerations or discussions of distributed query surveillance structures?
MR. BUELLER: So just in case others aren't familiar with that concept, the notion of a distributed system is that the data sits behind each owner's firewall and when you have a question you develop a query and the data owners are asked to hold the data again by format so that you can craft the query, go out, hit that against each of the data owners, and bring back aggregate report.
We tried something like that, there is actually a system called Distribute that was a grass roots system, developed in the 1990s. When H1N1 hit, we put a lot of effort into scaling that up. And basically rather than getting individual level data we're asking the states that had syndromic systems to provide aggregate counts among a handful of variables.
And that worked reasonably well up to a point, and I know that it parallels in a small way, many of the discussions I've heard at FDA around something that they've got called, Sentinel, or things like the HMO Research Network. It takes a lot of work to understand what you're getting when you can't get it at the individual level, and it's much less flexible, particularly if something happens and you need to query the data in a way that you hadn't done it before.
It's much easier if you have that individual level data, particularly when you see something and then the first thing that happens is you start to get five more questions, and to be able to answer those questions is much easier at an individual level.
But with that said, there is a lot of precedent, the CDC, many of these things like the HMO research network which is the forebearer of the FDA Sentinel was based on a project that really came out of CDC with the vaccine safety data language that's based on this notion of a distributed approach. That has been used, VSD, Vaccine Safety Datalink, is a very successful project, but it's like any approach, it has advantages and disadvantages.
I know it's one that ONC is very keen on supporting, the Office of National Coordinator for Health Information Technology, they've got a query health project and members of our staff have been involved in helping them think about that. So it's one approach, we do think about it, it has strengths and limitations.
DR. CARR: Thanks. That was very helpful, very informative, we appreciate it very much. And you're welcome to stay on the line, we're just going to have a couple of concluding comments from today's meeting. Thank you for joining us.
Agenda Item: Wrap up for Future Meeting Next Steps
DR. CARR: I want to try to pull together what we've covered today, in the last three hours, and put it out there for your consideration and ask for any suggestions or other issues.
I would say where we are today is that we can say that it is good that HHS has liberated the data. We all agree that that availability is tremendous opportunity. Second, that we have seen interesting examples from challenges from the Datapalooza, demonstrating new observations that can be drawn from the data alone, or from merging the data from multiple sets.
We also observe that the use of data, at least based on the hits on the website that we've seen today, the use seems modest and could be higher, and that the challenges toward higher use are three. One is knowledge of the data availability. The second may be the usability of the data, the ease of use of using it. And the third might also take into account formulation of priority issues that ought to be addressed that would drive someone to those data sets.
One other observation is that there is a very strong intersection between the issues we've been addressing in the working group and the full committee focus on empowering communities to access and use data, and we're going to make sure that everybody has a copy of this report.
But just to read briefly from the report and the executive summary of what, based on the hearings that we had, what the communities said that we needed. A key need was infrastructure to provide support, facilitate shared learning, and create economies of scale.
And specifically, they felt important components of the infrastructure include a privacy and security framework to guide communities in using local data. A standardized set of community health indicators. Training and technical assistance, to improve data access, management, and analysis methods and competency. Better data visualization tools and skills, something we talked about today. Support and external facilitation to strengthen local financial human resources, including those for coalition development. Guidance on achieving data informed approvement through effective leadership and mechanisms to enable communities to share knowledge and information, stay abreast of federal and state resources and activities.
And then in the section on envisioning the federal role, there are a number of recommendations. I'll just read six of them. NCVHS has identified ways in which the federal government can support the development and functioning of community base or community oriented learning systems.
One is facilitate and provide resources to strengthen communities' capacity to collect data. Drawing on health indicators warehouse, continue to identify and encourage adoption of standardized community health indicators.
Three, provide local communities with local data on environmental resource factors including economic housing, transportation, and education data that are routinely generated by state and federal entities. Four, promote development use of federal and state web-based query systems to provide small area data, easy analytics ,and visualization capabilities.
Five, expand technical assistance mentoring communities in survey design, data collection, data analysis, et cetera, and convene a summit of local communities to share what they're doing and enumerate a set of barriers that effect all communities working to improve local health.
So clearly there's a tremendous intersection about what we've been talking about today, how to address some of these issues, and these are the issues that we heard in the hearing. So I'd like to suggest, again, that when we meet again, whether it's in person or by phone, but we will certainly be meeting at the next February NCVHS – March 1st, okay.
And at that time I think in order for us to be a reactor panel, I think it might be helpful for two things: one is to get an update from HHS on what they are already contemplating, what they have considered and accomplished or considered and rejected, and why. And then actually to have some of those data folks from HHS come to our next meeting so they can have a dialogue. Because I think that's the way that we can communicate quickly as we did when Todd convened the committee and we just had a meeting, there were no recommendations, we gave reactions and went from there. So I think that's a data reactor kind of venue.
And I think some of the things that we've covered today, and we'll explore more what Lee was suggesting we could bring to that group. And then I think we need to probably marry up the work that could be done by this group and the work of the full committee, on empowering communities, as we said take a copy of this report because I think it very much addresses the issues we have.
So with that summary I'd like to invite any additional comments, suggestions, recommendations.
DR. COHEN: The one additional piece would be I'd like us to explore additional web access technologies to learn more about those that aren't necessarily HHS or government-oriented, that use health data or other vehicles that haven't used health data yet.
DR. CARR: Now, I think what I am going to do is confer with Marjorie and Jim – oh, Vicki.
DR. MAYS: I was just going to ask, is the committee interested in other vehicles besides the web? Because there are partnerships that people are doing with like Wal-Mart's and Walgreen's where they have video screens and they're looking for health messages. There is a big movement right now in the drug stores and they're looking to get the health information to it, so I just want to put it on your radar.
DR. CARR: My thought would be that we stay grounded in what our charge was, and I think that the prominent in that as I read it today and I read it before, is the feedback to HHS on what they can do to get this out. Well, I heard the charge is grounded in the data, the usability of the data, the knowledge about the data and kind of priority issues.
So what you suggest might be something that we take up more in the community data initiative. But I think for now we're still trying to get our hands around getting the feedback configured properly to HHS.
DR. VAUGHAN: Susan just whispered in my ear and it amplifies with what Vickie was just saying, but it's an excellent point. Looking at that as maybe a larger subset of what's called mHealth or mobile health, or kind of reaching folks with good health information, including data, these alternative systems.
DR. CARR: It is pushing out data?
DR. VAUGHAN: Yes. One of the most interesting, for me, is text for baby which pushes out pregnancy wellness data based on gestation. So that has been immensely successful, very low cost, and is being replicated across the agency and other instances. So it doesn't always have to be something that's expensive or fancy, sometimes it's just going to where the community is and what they can use and putting it in a framework that they can use it.
DR. GREEN: Justine, question about scope again, and staying in the charge. But where do devices like asthma inhalers with geospatial devices, do they fit into the charge or is it really just internet?
DR. CARR: I know Jim's priority is getting the feedback on the HHS data. I think that the asthma obelisk and so on, I mean obviously we're going to intersect, in fact the call right now about the Datapalooza is going on.
But I think this committee, even though we're asynchronous, we're not quite where we want to be, we're still in the learning curve, I think the amount of learning that we've had these last couple of sessions can inform. And perhaps we ought to be thinking about for the Datapalooza, back to Josh's point, of how do you incentivize specifically community data or something like that for that interception.
I think what I'm going to do, Susan Queen and I, I think we'll arrange a conference call and/or webinar, certainly before February, probably early January. But the other thing is we'll get the summary of this meeting out to everyone and invite input for the folks who were unable to be here today.
So I think with that we'll conclude and adjourn, and I thank you all for coming and wish you a safe travel home. Thank you.
(Whereupon, the meeting adjourned at 5:10)