I truly love my thesis topic and am so happy I chose it. However, it is causing great difficulties as I avoid all my other work because I am having so much fun. I have completely messed up my sleep schedule because I get home, start checking OCR’s and looking for codebooks, then suddenly it’s 3:00 AM. I have been searching for every code book under the sun, and have found at least 10 books that were not used by NARA. Also, I currently theorize that NARA’s focus on obtaining codebooks, rather than the changes in punch card regulations, was a major contributor to their decoding errors. I am using ABBYY fine reader to use OCR on the codebooks included in NARA’s documentation. This is taking far longer than expected, mainly due to poor scan qua, however it is also incredibly fun. The challenge is that it is very difficult to stop because I want to keep going through the books. Thus far I have extracted about 200 pages - the counties and states for 1944 (3,095 rows) and station codes for 1945 (I have not finished this yet but I estimate 2,000 rows).
Additionally, the Library of Congress holds a complete copy of the 1942 and 1943 codebooks, which would be highly useful to see, but they are each around 500 pgs and not currently digitized. I am trying to determine a cost effective way to obtain copies.
So far I have learned quite a lot about military manuals and the Adjunct Generals Office. HathiTrust has proved the most useful. The University of California hosts the largest collection of Technical Manuals I can find (of course they are missing the main one I need). I have also learned that despite being 80 years old, you must have a CAC to access the TM ‘s from the 1940’s if you go through the Army publication site. They also have some TM s marked as active from the 1930s - however they are for series 9 (Ordinance Dept) not series 12 (AGO). For context, I am looking at TM 12-310 and 12-305. I think the AR reports should be looked into for coding processes, however (much to my dismay) there are so many possible numbers to look at it is not feasible for this project.
As I write this, I am coming to terms with the fact that I must stop seeking out additional codebooks due to time. However, I am still determined to obtain the LoC TM 12-310 for personal interest. If I am able to, I will keep a version on my computer, make copies of at least two dedicated hard drives to put away for safe keeping in separate locations, and print a copy, for which I will order a perfectly sized box from Gaylord. I have looked at every website, database, catalog, and retailer I can find that has anything to do with WWII, the Army, and technical manuals, but I have not been able to find a complete copy - there isn’t even a forged insanely overpriced copy on EBay or AbeBooks.
The ‘actual’ challenge has been the literature review. This is not due to lack of material, rather that my sources struggle to communicate. I am starting from the beginning. As there is always the possibility that I missed something, am using the wrong search terms, or am looking in the wrong places, I hesitate to say there is no research on processing intrinsic big data or extracting the spatial aspects of said data. What I can say is following hours upon hours of research, I can’t find anything on the topic. So, I found a wide variety of data handling practices from archeology, machine learning, and cultural heritage. Essentially, I am looking at these sources, and picking aspects of each to develop a workflow. Perhaps this is paradoxical as by creating a workflow from multiple sources I am creating a commonality and thus they do communicate. However, this does make their communication quite difficult.
I decided to add a “Data Review” as I realized in many ways the most critical part of this project is understanding the data I am working with. This may sound simple but the data is in no way clean cut. In fact it’s a complete mess and everything conflicts. The more I learn about the data, my shock that the enlistment file database is available for public use increases exponentially.
I do think this again raises a question which I continue to mull over. The enlistment file is one of the most used, most critical, and most relevant archival records set for genealogists. Yet every time you search for a record, there is at least a 74% likelihood that a record you are looking at is inaccurate. Websites such as ancestry.com pull data from AAD, increasing the output of misinformation. However, is it better to have your great grandfather's rank spelled “p0t” instead of “pvt” or not have it online at all?
Personally, I feel like it should remain available despite its inaccuracy, however it needs to be plastered all over the website that the records contain a high degree of inaccuracy, and if you don’t find someone, it doesn’t mean they aren’t there or the information is correct.
Overall, I think the most remarkable aspect is that there is no active effort whatsoever to reprocess the records. In some ways this research is pointless, because without accurate images there is no way to check digitization accuracy. The microfilm needs to be reprocessed and reanalyzed - it’s been 25 years since anyone used the original microfilm. However perhaps the point of this research is to show what could be possible if the records were reimaged.
No comments:
Post a Comment