As of right now I am just going to add this to my own to-do list as those who were interested have informed me of how time-consuming this is going to be. Since the Compendium will take so long to do anyway, I'm not worried about getting any particular part of it done quickly at the moment.
My decided process is to search each World News article by a set of keywords. If any of the following keywords are included a news article, they will be included in the News section of compendium regardless of the usefulness of the actual mention:
Caldari
Provist
CEP
Sukuuvestaa
Nugoeihuvi
CBD
Ishukone
Hyasyoda
Kaalakiota
Wiyrkomi
Lai Dai
I can't think of a situation where a State-related news article would not have at least one of those keywords. If anyone can think of an example that would suggest I add a keyword to that list, please let me know.
Depending on your time/ skills/ resources, you could probably hire some newbies as in game, in character 'private detectives' to find this stuff out for you.
Pay them as much money to do research as they would make mining, and give them an IC reason for researching this. You could tell them that you are a private entity subcontracted with the
Corporate Police Force.
Newbies are eager to help out, learn the game, AND eager to make money. Despite this, when they are starting out they barely have enough skills in game to be able to fly a hauler. They can't 'jump right in', and that can be frustrating. Telling them that they can be a Policeman/ Detective makes them feel useful/ important from day one, and 90% of 'undercover field work' involves observation.
What you are asking requires no actual in game skills, just out of game research skills. This is a way that eager newbies can be helping the corp out without actually even playing EVE. They also feel like an integral, important part of the game world.
Private detective/ reporter would probably be the best IC roleplay reason. Just use some sort of formula for writing murder mystery stories to develop an 'end game' plot, such as "It was Miss Marple in the study with the candlestick."
For example, what connection would Provists and the Hyasyoda corporation have? Aren't they from politically different factions? Who murdered Tibus Heth?
You have eleven topics mentioned here. It would be more fun/ practical to get eleven newbies/ employees each doing in depth research on one assigned topic, rather than asking one person to do it all.
Throw in some missions as part of the assignment as well, such as hauler missions to increase corporate standings with Lai Dei to farm datacores. Missions for Caldari Navy would help grind standings for Jita station trading taxes, too.
Contract 'livestock' (if that is possible) as part of the 'character cast' in your storyline. Exotic dancers, marines, Scientists, janitors can be prisoners, passengers, VIPs, criminal suspects, witnesses and so on. Every character would have a different story crafted by you. Every commodity item becomes a clue, another piece of the bigger puzzle.
Get your low skilled employees to haul skillbooks in shuttles and sell them in Jita as a 'cover front'. That way, they can earn ISK to make themselves self-reliant.
Eventually, your employees will probably be working their way to train up to covert ops frigates for surveillance purposes. Tracking down a high sec enemy of your choosing with locator agents would be fun too.
High sec war decs would just seem like a plot climax in the story. They have the coincidental appearance that 'Corporation X' are attacking because they must be involved in the plot somehow. "Why are they attacking us? Hmm. Does someone have something to hide?"
PVP training would be an essential part of being a private detective of course. If they decide they don't want to be 'private detectives', the cov ops skill are transferable to Faction war scouts, intel channels and so on.