WARChain: Consensus-based trust in web archives via proof-of-stake blockchain technology 1

Abstract

Web archives store born-digital documents, which are usually collected from the Internet by crawlers and stored in the Web Archive (WARC) format. The trustworthiness and integrity of web archives is still an open challenge, especially in the news portal domain, which face additional challenges of censorship even in democratic societies. The aim of this paper is to present a light-weight, blockchain-based solution for web archive validation, which would ensure that documents retrieved by crawlers are authentic for many years to come. We developed our archive validation solution as an extension and continuation of our work in web crawler development mainly targeting news portals. The system is designed as an overlay over a blockchain with a proof-of-stake (PoS) distributed consensus algorithm. PoS was chosen due to its lower ecological footprint compared to proof-of-work solutions (e.g. Bitcoin) and lower expected investment in computing infrastructure. We based our prototype on the open-source Nxt blockchain and implemented it in Python. The prototype was tested on web archive content crawled from Hungarian news portals at two different timestamps with more than 1 million articles in total. We concluded that the proposed solution is accessible, usable by different stakeholders to validate crawled content, deployable on cheap commodity hardware, tackles the archive integrity challenge and is capable to efficiently manage duplicate documents.

Keywords

Web archive validation blockchain proof-of-stake web crawling censorship

Get full access to this article

View all access options for this article.

References

14873 ISO/DTR, Information and documentation – Statistics and Quality Indicators for Web Archiving, https://netpreserve.org/resources/IIPC_project-SO_TR_14873__E__2012-10-02_DRAFT.pdf, last accessed 2021/02/28.

Alam,

Kelly,

M.C.

Weigle and

M.L.

Nelson, Unobtrusive and extensible archival replay banners using custom elements, in: Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries, IEEE, 2018, pp. 319–320. doi:10.1145/3197026.3203881.

Alam,

Weigle,

Nelson,

Melo,

Bicho and

Gomes, MementoMap framework for flexible and adaptive web archive profiling, in: 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL), IEEE, 2019, pp. 172–181. doi:10.1109/JCDL.2019.00033.

Alam,

M.C.

Weigle,

M.L.

Nelson,

Klein and

Van de Sompel, Supporting Web Archiving via Web Packaging, 2019, arXiv preprint arXiv:1906.07104.

Aturban,

Alam,

M.L.

Nelson and

M.C.

Weigle, Archive assisted archival fixity verification framework, in: Proceedings of the 19th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’19), ACM, 2019, pp. 162–171.

Barbaresi, in: Generic Web Content Extraction with Open-Source Software. Conference on Natural Language Processing (“Konferenz zur Verarbeitung Natürlicher Sprache”, KONVENS 2019), Erlangen, Germany, 2019, pp. 267–268.

D.J.

Bodeau,

C.D.

McCollum and

D.B.

Fox, Cyber Threat Modeling: Survey, Assessment, and Representative Framework, Homeland Security Systems Engineering & Development Institute, 2018.

Butler, Addressing Recent Claims of “Manipulated” Blog Posts in the Wayback Machine, http://blog.archive.org/2018/04/24/addressing-recent-claimsof-manipulated-blog-posts-in-the-wayback-machine, last accessed 2020/07/08.

Castells, The Rise of the Network Society, John Wiley & Sons, 2012.

10.

Collomosse,

Bui, Brown et al., ARCHANGEL: Trusted archives of digital public documents, in: Proceedings of the ACM Symposium on Document Engineering 2018, ACM, 2018, pp. 1–4.

11.

Gomes,

Miranda and

Costa, A survey on web archiving initiatives, in: International Conference on Theory and Practice of Digital Libraries, Springer, Berlin, Heidelberg, 2011, pp. 408–420.

12.

Holzmann,

Goel and

Anand, Archivespark: Efficient web archive access, extraction and derivation, in: Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries, ACM, 2016, pp. 83–92. doi:10.1145/2910896.2910902.

13.

Indig,

Kákonyi and

Novák, Crawling in reverse – lightweight targeted crawling of news portals, in: Proceedings of the 9th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics,

Kubis, ed., Wydawnictwo Nauka i Innowacje, Poznan, Poland, 2019, pp. 81–87.

14.

Indig,

Á.

Knap,

Sárközi-Lindner,

Timári and

Palkó, The ELTE.DH pilot corpus – creating a handcrafted GigawordWeb corpus with metadata, in: Proceedings of the 12th Web as Corpus Workshop, Marseille, France, 2020.

15.

ISO 28500, WARC (Web ARChive) file format, http://www.digitalpreservation.gov/formats/fdd/fdd000236.shtml, last accessed, 2020/05/15.

16.

Johnson and

Thomas, Interfaces with the past...present and future? Scale and scope: The implications of size and structure for the digital archive of tomorrow, in: Proc. Digital Heritage Conf, 2013.

17.

Kelly,

Alam,

M.L.

Nelson and

M.C.

Weigle, InterPlanetary wayback: Peer-to-peer permanence of web archives, in: International Conference on Theory and Practice of Digital Libraries, Springer, Cham, 2016, pp. 411–416.

18.

Kelly and

M.C.

Weigle, WARCreate: Create wayback-consumable WARC files from any webpage, in: Proceedings of the 12th ACM/IEEE-CS Joint Conference on Digital Li-Braries, ACM, 2012, pp. 437–438. doi:10.1145/2232817.2232930.

19.

Khullar, Implementing Proof of Stake Part – 2, Medium.com, https://medium.com/coinmonks/implementing-proof-of-stake-part-2-748156d5c85e, January 2019.

20.

Laurie,

Langley and

Kasper, Certificate Transparency, Internet RFC 6962, 2013, https://tools.ietf.org/html/rfc6962.

21.

V.L.

Lemieux, Blockchain Technology for Recordkeeping: Help or Hype? Technical Report, U. British Columbia, 2016.

22.

Lendák,

Indig and

Palkó, WARChain: Blockchain-based validation ofweb archives, in: 10th International Workshop on Socio-Technical Aspects in Security, Guildford, United Kingdom, 2020.

23.

Lerner,

Kohno and

Roesner, Rewriting history: Changing the archived web from the present, in: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Com-Munications Security (CCS 2017), ACM, 2017, pp. 1741–1755. doi:10.1145/3133956.3134042.

24.

Milligan, Lost in the infinite archive: The promise and pitfalls of web archives, International Journal of Humanities and Arts Computing10(1) (2016), 78–94. doi:10.3366/ijhac.2016.0161.

25.

Milligan, Welcome to the web: The online community of GeoCities during the early years of the World Wide Web, in: The Web as History: Using Web Archives to Understand the Past and the Present,

Brügger and

Schroeder, eds, UCL Press, 2017, pp. 137–158.

26.

Münster, Digital heritage as a scholarly field – topics, researchers, and perspectives from a bibliometric point of view, Journal on Computing and Cultural Heritage (JOCCH)12(3) (2019), 1–27. doi:10.1145/3310012.

27.

Pomikalek, Removing boilerplate and duplicate content from web corpora, Ph.D. thesis, Masaryk University, Faculty of Informatics, Brno, Czech Republic, 2011.

28.

Schafer and

Winters, The values of web archives, in: International Journal of Digital Humanities, 2021, pp. 1–16. doi:10.1007/s42803-021-00037-0.

29.

Schreibman,

Siemens and

Unsworth, A Companion to Digital Humanities, John Wiley & Sons, 2008.

30.

Scwartz, Ethereum 2.0: A Complete Guide. Casper and the Beacon Chain, Medium.com, https://medium.com/chainsafe-systems/ethereum-2-0-a-complete-guide-casper-and-the-beacon-chain-be95129fc6c1, last accessed, 2020/07/08.

31.

Shatnawi,

Munson and

Thao, Maintaining integrity and non-repudiation in secure offline documents, in: Proc. Intl. Conf. on Document Engineering, ACM, 2017.

32.

Shostack, Threat Modeling: Designing for Security, Wiley, 2014.

33.

Sigurðsson, Managing duplicates across sequential crawls, in: Proceedings of the 6th International Web Archiving Workshop (IWAW, 2006.

34.

Van de Sompel,

M.L.

Nelson and

Sanderson, HTTP framework for time-based access to resource states–Memento, RFC 7089, 2013.

35.

Vismann

translated by Winthrop-Young, G., Law and Media Technology, Stanford University Press, Stanford, CA, 2008.

36.

W3C, An Overview of the PROV Family of Documents, https://www.w3.org/TR/prov-overview, last accessed 2020/07/08.

37.

Walport, Distributed Ledger: Beyond Blockchain, Technical Report, UK Government, 2015.

38.

Webster, Users, technologies, organisations: Towards a cultural history of world web archiving, in: Web 25, Histories from 25 Years of the World Wide Web, 2017, pp. 179–190.

39.

Winters, Coda: Web Archives for Humanities Research Some Reflections, UCL Press, 2017, p. 238248.

40.

Winters, Breaking in to the mainstream: Demonstrating the value of Internet (and web) histories, in: Internet Histories 1.1-2, 2017, pp. 173–179.

41.

Yasskin, Web Packaging, https://github.com/WICG/webpackage, last accessed 2020/07/08.

42.

L.L.

You,

K.T.

Pollack and

D.D.

Long, Deep store: An archival storage system architecture, in: 21st International Conference on Data Engineering (ICDE’05), IEEE, 2005, pp. 804–815. doi:10.1109/ICDE.2005.47.