Outline: 3 Steps

 

 

Hello Justin,

What I want from you:

I’d like your feedback on an App I developed to provide readers with more Context.

Is there an Audience for more Context?

At the end of December, I saw Beau’s forecast that declining mainstream media will sensationalize the news in the coming next year.

In contrast, YouTube news channels that provide context are growing.

Adding Context to the Web:

For the last few years I’ve been developing an app to enable responsible authors to show the context of their web and video quotations. My local newspaper has not shown interest in the app (even though its free), but I’m hopeful that independent media may.

3-Minute Video:

Nothing is more certainly written in the book of fate than that these people are to be free.

• View CiteIt website:

Note:

• I’m not selling software. (The App is free and open source).

Feedback Questions:

1. Do you think there is an interest among readers and publishers for more context?
2. What do you think are the biggest obstacles to adoption of CiteIt-style Contextual Citations?
3. Do you know of other people who might be interested in this feature?

Beau Examples:

• View Beau example videos using the videos’ Youtube Transcripts:

 

“Direct and Easy” Citations vs “Do Your Own Research?”

I know that in Beau’s videos he prefers not to link to sources, but rather have viewers look up sources themselves. I’m curious on your thoughts about direct quotes in web-based “print” journalism, like Greedmedia or Propublica articles that have direct quotes from their own interviews or that link to many of their sources.

• If there is an interest in developing Contextual Citations further, I would love to work with creators in Independent Media.

 

About Me:

• I’m a History major who has worked as a computer programmer for the last 20 years.
• As a hobby, I’ve used my programming skills to develop an App called CiteIt to help responsible writers contextualize their sources.

 

Tim Langeman
Akron, PA (USA)
717-723-9898
timlangeman@gmail.com
https://www.openpolitics.com/tim (home page)
https://www.citeit.net (hobby project)

There is an excellent website called Oyez that contains interactive transcripts of Supreme Court hearings which allow readers to click on a line in the transcript to play the audio of the text.

Interactive Transcript

Citizens United vs FEC: Oral Re-argument – Sept 9, 2009

The transcript is generated from a JSON file which contains start and end times and labels each speaker.  You can view the rendered user interface, but if you view the source you will find that it does not contain the transcript text.

Because the transcript text is derived from a JSON data file rather than static text or html, CiteIt is unable to read the transcript text.

Locating Case Identifier

It would be nice if there was a way to generate a text version of the transcript so I wrote some Python code to convert the JSON file to a text/html file.

It is possible to use CiteIt to quote from the Supreme Court transcript, if you know the URL of the JSON file.

You can construct the transcript JSON URL by inspecting the generated HTML.

Oyez JSON file

Oyez JSON URLs follow the following URL format:

• • URL: https://api.oyez.org/case_media/oral_argument_audio/22476
  • View JSON:

Generating Transcript from Oyez JSON data

You can generate the text version of the transcipt using the webserivce:

Generated Oyez Text Transcript

The generated Oyez transcript looks like this:

• http://api.citeit.net/v0.4/url/text-version?url=https%3A%2F%2Fapi.oyez.org%2Fcase_media%2Foral_argument_audio%2F22476&line_separator=

Quoting Oyez Transcripts Using CiteIt

Then you can then use CiteIt to extract the context to contextualize the quote:

<blockquote cite=”https://api.oyez.org/case_media/oral_argument_audio/22476“>

<p>The government silences a corporate objector, and those corporations may have the most knowledge of this on the subject.</p>

<p>Corporations have lots of knowledge about environment, transportation issues, and you are silencing them during the election.</p>

</blockquote>

 

Result:

The government silences a corporate objector, and those corporations may have the most knowledge of this on the subject.

Corporations have lots of knowledge about environment, transportation issues, and you are silencing them during the election.

 

 

 

 

2 Ways of adding Context when Quoting Beau’s Videos:

 

1) Expanding Blockquote:

Beau reports that media traffic numbers are down from 2020.

As a result, he predicts more sensationalism in 2022.

In an attempt to draw the same viewership as 2020 media are going to have to:

give you something sensational to latch on to. Stories that they can milk and keep in the news cycle. That’s what I’ve got my money on. I would be prepared for a lot of sensationalist content to start coming out

2) Inline Contextual Popup:

↓  Click on grey-blue link to view video popup  ↓

In another video, Beau suggests that the problem in Afghanistan was that the US wanted to keep Afghanistan as a weak client state— We wanted the national government in Afghanistan to be just strong enough to do what we told them.

 

 

About CiteIt:

The above quotes were copied from the auto-generated YouTube transcript and linked with the CiteIt WordPress plugin.  The app automatically pulled in the context above and below and embedded the video.

You can see an more video examples on the home page.

• Quoting YouTube Transcripts with CiteIt.net (video tutorial)

 

CiteIt’s Benefits:

1. Like the basic ability to embed pictures, audio, and video, CiteIt enables writers to increase the reader’s contextual understanding of an issue with minimal effort.
2. Readers benefit from greater understanding of the material and confidence in writers
3. Publishers can differentiate themselves from the competition and benefit from reader’s greater trust and loyalty. (speculative).

 

 

1) When Context is stripped away: David Wolpe

Tyler Cowen‘s interview with Rabbi David Wolpe is interesting because it suggests what is lost in social media:

But there’s something about social media that seems to act to strip away context, and people who write for mainstream media will tell you this: “Well, I wrote an article that ended up on Facebook in a very different setting than how I intended it to be read.” And you can say all you want — all the hyperlinks are there, but people don’t click through.

What do you think is the intellectual future of a belief system based on commentary on commentary on commentary, now injected into a world with this technology that so strips away context and just gives you some bald statement of something?

WOLPE: I think that Judaism has the same problem that any thick civilization has in a world in which, as you say, context is stripped away. And not only is context stripped away, but attention to any one thing is scanter and less than it used to be.

 

Note: This YouTube Video doesn’t have a Transcript

Even though the “Conversations with Tyler” Medium website does include a transcript, unfortunately, the YouTube video doesn’t.

To make this demo work, I had to pretend that the videos have a transcript and hard-code this YouTube Transcript.

2) Contextual Video Popup from Transcript: Malcolm Gladwell

Many YouTube videos do have transcripts.

In a speech at the Miami Herbert Business school, Malcolm Gladwell discussed the psychological and social implications of real estate.

↓ Click grey-blue link below to View Popup

One point Gladwell made was that there are no bad neighborhoods there only bad blocks. By this Gladwell meant that crime is concentrated in small areas (blocks), which require a concentrated response, rather than a blanket “stop and frisk” policy that affects whole neighborhoods.

 

 

3) Howto create Contextual Video Citations using WordPress

a) I’ve created a WordPress plugin which enables writers to create this form of contextual citation by linking their quotations to a source by using a “cite” URL.

b) When the post is published, the WordPress plugin automatically calls my webservice to lookup the source context.

c) It also embeds the YouTube video using the “cite” URL.

 

Sample HTML:

<q cite="https://youtu.be/cE1g3GJCq7o?t=2587">
there are no bad neighborhoods there only bad blocks
</q>

(you can specify the video’s start time using the t=seconds syntax)

 

Read More

• Read more about how to use CiteIt to create Contextual Audio/Video Citations

How did CiteIt come about?

Learning HTML in College

During college, I took an interest in teaching myself Html.  When I first took Computer Science classes, the University did not teach Html.  That was something I did myself using Html tutorials and inspecting the source of pages.

My residency had a rudimentary network on which we used a Windows network share to share files.  I took it upon myself to create a web version of the Cafeteria Menu that was popular and I also created an Html version of the Student Council finances, as I was student council treasurer.

I recall thinking how much more quickly research could be done if I could search the library from my dorm room and if book citations could be hyperlinked together.

Ted Nelson: Hypertext Pioneer

My online research into hypertext led me to hypertext pioneer Ted Nelson who coined the term “hypertext” and who worked to design, build, and popularize the idea of hypermedia in the 1960s, 70s and 80s, before Tim Berners-Lee introduced a “compromised” type of hypertext in the World Wide Web.

I read as many of Nelson’s books as I could get a hold of and when I had one publication remaining — his Ph.D. thesis written in Japan — I wrote to Japan and then later to Nelson himself to track down a copy.

Making Amazing Connections

One of the things I have learned throughout my CiteIt project is that it is possible to make some amazing connections that many people who think impossible, so long as the accomplished person is not famous.  Ted Nelson is “famous” within the hypertext community but he doesn’t receive the influx of traffic that Bill Gates does.  I wrote to him, with nothing to lose, and he wrote back!  He even included scanned Pdf copies of his thesis!

Ted’s Thesis

I wanted to impress Ted, so I got myself a license to Adobe Acrobat and used its OCR feature to convert the scans to Microsoft Word (yes, I know Ted hates MS Word).  I helped convert his thesis to something he could sell on his website.  I even found a few typos, which Ted realized I could of only have caught with the help of spelling and grammar check.  I sent him the digital copy of the PDF and the Microsoft Word document.

I wanted others to understand who Ted Nelson is and what he had done, so I started writing a review of his Ph.D thesis.  As I was citing sources in my review I got to thinking that there ought to be some way of quoting from the thesis that was more like Nelson advocated in his original vision.

My Review of Ted Nelson’s Thesis

• Ted Nelson’s Philosophy of Hypertext

 

 

Technologies Learned

Combine Html knowlege from WordPress: <blockquote cite=”URL”>

Use knowledge of Beautiful Soup

Learn how to Create custom jQuery function

Learn how to Create WordPress Plugin

Start with a php version

Switch to Django

Switch to Flask

Learn about sha256 hashing

Learn about Unicode

Learn about Javascript Utf-16

Learn about Amazon S3

Learn about creating Docker Image from python

Started to use SQL Alchemy

 

 

In the 1960s, Ted Nelson envisioned a universal hypertext system called Xanadu that was much more sophisticated than today’s web.  Xanadu was carefully designed to achieve the entirety of Nelson’s vision in an elegant way.  Xanadu was what Richard P. Gabriel would call The Right Thing — a project that doesn’t compromise its design in the areas of Simplicity, Correctness, Consistency, or Completeness.  However, as Gabriel observed in The Rise of “Worse is Better“, “The Right Thing” often loses the competition for adoptiono to a design that prioritizes simplicity over consistency and completeness.

In 1990, when Gabriel wrote “The Rise of Worse is Better“, he was a Lisp programmer engaging in some self-criticism about the failure of Lisp and the MIT philosophy to gain market adoption verses C++ and Unix, but his argument could apply equally to Xanadu’s relationship with the web.

In his follow-up articlee, “Is Worse Really Better?“, Gabriel argued:

Worse-is-better programs are like viruses that spread quickly and are soon pervasive. And over time, if they are successful, they will be improved. The argument is an evolutionary one—a system that establishes a large territory and molds its environment before rivals show up has the advantage over those rivals.

The web’s dramatic growth certainly meets Gabriel’s standard for viral spread. The question remains as to whether the compromises the web made will ever actually be improved upon to the point that it approaches the “Right Thing”:

Once the virus has spread, there will be pressure to improve it, possibly by increasing its functionality closer to 90%, but users have already been conditioned to accept worse than the right thing. Therefore, the worse-is-better software first will gain acceptance, second will condition its users to expect less, and third will be improved to a point that is almost the right thing.

Nelson’s design for Xanadu prioritized designing the correct back-end system with the goal of enabling others to develop the desired front-end functionality.  By contrast, as Jeff Atwood describes it, the web developed with the absolute minimal architecture necessary:

the most fundamental building block of the web, the hyperlink, barely works at all. Hyperlinks are fraught with peril and pitfalls even under the best of conditions. The current state of hyperlinking is almost literally the stupidest thing we could build that works.

What’s still missing from the web

The web is still missing 4 main features proposed by Xanadu.  It is not possible to graft these features onto the web using the uncompromising principles of “The Right Thing”, however it is now possible to design an application for web authors that adds the first two missing features —

1. two-way links and
2. contextual citations.

Advances in cryptocurrencies make some approximation of the third feature — micropayments — foreseeable in the medium-term, while the fourth feature — transclusion — is likely only fully possible after a completely new back-end is implemented that does not use html’s embedded markup.  However, a “quick and dirty” implementation that mimics transclusion and parallel quotation is possible in the mid-term.

Bridging towards our Potential

My CiteIt.net Contextual Citations are a partial implementation of parallel documents that can bridge far enough towards Nelson’s vision of parallel documents for the public to see contextual citations in action.  My hope is that once the public is able to see what is possible, they will begin to demand a more full implementation of parallel documents and entrepreneurial programmers will be inspired to develop such designs.

4 missing features

Features that are possible now are listed below in green.  Orange is used to designate mid-term features and red is used for long-term goals:

1. Two-way Links/Fix Link rot
2. Contextual Citations/Parallel Documents
3. Micropayments
4. True transclusion

The Importance of Business Strategy

While the individual features are technologically feasible, it will take more than just technology for hypertext to realize its potential.  It is of equal importance that a business strategy be found to generate the funding to develop and maintain the technologies in an application that is adopted and by a sizable market of users.

 

One such strategy could combine two existing projects into a single app with the following two features:

1. Link rot : Wayback machine:
2. Two-way Links /Contextual Citations : CiteIt.net
   

Showcase Project: Wikipedia

1) Internet Archive

To gain customer interest for an app, a showcase project could be used to demonstrate how such a system could work on a large enough scale.

The Internet Archive currently partners with Wikipedia to fix linkrot using its Wayback machine. The Internet Archive also has plans to implement Contextual-like Citations for Wikipedia sources.

2) CiteIt.net

Contextual Citations, as implemented by CiteIt.net in this document, are a “bridging”-technology for bridging from tranditional Html links to fully parallel links as envisioned by Ted Nelson. Although CiteIt.net’s Contextual Citations do not display both sources in parallel, they, along with something like Google’s Text Fragments allow the author to specify the exact quote, even if the document contains multiple instances of the quote.  With CiteIt, Website owners can choose whether to display the context using a popup window or  a contextual div above or below the quote.  If sufficient adoption is achieved, browser performance and display could be optimized through a plugin or native functionality.

In the immediate term, CiteIt.net offers a WordPress plugin for the old classic pre-Gutenberg editor.  This app and webservice could be extended and integrated into The Internet Archive’s Wikipedia project.  This would require a fair bit of work to make the features production ready, but when these two features are achieved, they could serve as an attractive showcase that advertises two of hypertext’s missing features.

Automatic Partnership

Bringing these features to market on one’s own would be a challenge, so it may be useful to partner with a company like Automattic, which makes WordPress‘s Jetpack and Anti-spam app Akismet.

The benefit to Automatic and Jetpack of adding these features is that there are likely a significant number of WordPress authors who would appreciate the these features:

1. First, the app would have the ability to save an archived copy of every page authors link to.  An app that would submit these links to the Wayback machine would enable authors to create permanent links to their sources, even if the page has been taken down.
2. 

   Google Text Fragments

   The second part of the app would feature CiteIt.net’s Contextual Citations, which would make it easy for authors to create contextual citations that mimic the transclusion that Nelson advocates.  Present-day hyperlinked quotations only link to the page a quote is coming from, but tools like Google’s Text Fragments make it possible to specify the exact instance of the quote where a quote occurs multiple times. 1

Link Database

Another benefit of implementing Contextual Citations is that it would allow all citations to be voluntarily collected in a database that could be made widely available to developers and researchers.  This would allow a partial realization of one of Xanadu’s goals of having 2-way links that provide the reader a way to query to a “page’s” incoming links, not just view the outgoing links, as is customary now.   It currently takes the resources of enormous companies like Google and Microsoft to spider the web to determine a page’s incoming links.  A public link database would give smaller groups of programmers an opportunity to innovate with web link depiction and analysis and a could encourage authors to use semantic markup with their quotations.  Once the groundwork is laid, I can imagine many programmers creating their own ways of visualizing the links between pages in parallel.

KISS: Limited Scope

Gabriel hyperbolicly called his approach “Worse is Better” but another term for it could be KISS — Keep It Simple Stupid.  The strength of KISS philosophy is that it does a better job of avoiding the scope creep that sometimes afflicts “The Right Thing”.  Limiting the initial scope of this Hypertext Improvement Proposal to only the first two features has the potential to provide authors and readers with tangible benefits while avoiding getting bogged down in additional complexity.

Parallel Documents

Once the first two phases have been completed (phase 1) and there is an established base of documents, work could be done on phase 2 — to upgrade contextual citations to an implementation of parallel documents.  This could involve building out a javascript framework to load the linked sites within the browser window or it could involve adding support for parallel documents to the browser.  Browsers like Brave and Firefox may have an incentive to innovate where others are content to let others be first in this area.

Nelson provided a proof of concept demo (slow load) of what he envisions Xanadu could look like in in a web browser.  Once existing document quotations have been marked up, programmers could experiment to develop their own interfaces.

Here is a very basic screenshot of what contextual citations could look like if they simply combined

• CiteIt, with
• Browser-based popup, with
• Google’s Text Fragments

Converting the contextual popup to a native browser window would enable readers to scroll through the entire cited document.  A line connecting the citing quote to the cited document could be attempted later, as Nelson intended.

Future Browser Innovation

There are hundreds of Linux Window Managers.  It is my believe that browser innovation will happen once work is done to build out the authoring infrastructure and authors have a way to display context through basic popups.

Conclusion

Richard P. Gabriel predicted that minimalistic viral software like the web would outcompete “the right thing”.  Though the web hasn’t matured the way Gabriel predicted, the potential for something approaching Ted Nelson’s vision is still there with the right technology and business strategy.

The approach I’ve outline starts with the strategy of:

• limiting scope so that visible progress is achieved quickly
• starting with two existing projects:
  1. The Wayback Machine
  2. CiteIt.net
• building a showcase with Wikipedia to demonstrates what authors can achieve if they use the Automattic app.
• partnering with an organization like Automattic and/or Substack that has an existing user base to build Editor support for these features, along with a database of marked-up quotes that can serve as a base for future innovation

 

I’m open to other approaches.  Send me an email if you have ideas.

 

I’ve proposed that Wikipedia upgrade its quotes to contextual citations.

Summary:

This article describes what changes need to be undertaken to upgrade a Wikipedia page’s citations:

1. add the javascript and css to the head
2. wrap the quotes in <q> or <blockquote> tags
3. call the jQuery plugin in the footer
4. generate the contextual json data by calling the webservice

 

1) Add Javascript to the <head>:

Diff: original vs modfied

2) Wrap the quotes in <q> or <blockquote> tags:

Note: This step requires extracting the source’s URL from the reference.  I’m not sure the best way to do this programmatically.  I would start by trying pywikibot.

Code:

who imagines herself susceptible to attacks of tremors and palpitations (her "<q cite="https://www.gutenberg.org/files/1342/1342-h/1342-h.htm">poor nerves</q>") whenever things are not going her way

 Screenshot:

View Sample html: Pride and Prejudice

3) Call jQuery plugin in the footer:

Code:

<!-- ##### Begin: CiteIt.net Dependencies ##### -->
<div id='citeit_container'><!-- CiteIt-quote-context.js injects data returned from lookup in this hidden div --></div>
<script>
// Call CiteIt.net plugin on all q-tags and blockquotes:
$( document ).ready(function() {
jQuery('q, blockquote').quoteContext();
});
</script>
<!-- ##### End: CiteIt.net Dependencies ##### -->

 

4) Generate the contextual JSON data by calling the webservice

• You can run your own instance of the webservice, or
• Call the public webservice, as demonstrated by the sample code.

 

More Information:

• You can view this example’s Pride & Prejudice sample code: before and after files.
• GitHub code: 8 Sample Wikipedia articles
• Sample code: view | download

Here’s a new video describing how to use the WordPress plugin:

Get WordPress Plugin

 

Video Transcript (draft)

Hello, this is Tim Langeman, creator of CiteIt.net.

PROBLEM

1. Today with accusations about Fake News circulating widely, many readers
   suffer under a constant suspicion that any quotation they’ve read may have be taken out of context.
2. Other readers just want to know more about a quote’s context to better understand it’s meaning.

SOLUTION:

I’ve created the CiteIt.net App to address both reader concerns by creating a tool to enable
responsible authors to easily look up and display the contexts of their quotes.

Here is an example of the first type of Contextual Citation: the expanding blockquote:

1. notice the context expands when the reader clicks on an up or down arrow
2. And here’s the second type of contextual citation: the contextual popup which appears when the reader clicks on a quote.

BENEFITS

Authors who use the CiteIt.net App, give their readers more reason to trust them and distinguish themselves from media competitors.

DEMO

You can test-drive the CiteIt.net WordPress plugin by visiting demo.citeit.net

This will send you to the demo site’s WordPress login page.
Log in with the default password and you will be taken to the WordPress dashboard.

To make a test post, click “New -> Post”

Now, let’s create 2 new Contextual Citations from articles I found on Google News:

A) The first article has to with “bats” and is found on the BBC.

To create the citation:

1. We find and highlight a quote.
2. We copy the quote to our clipboard and paste it into to the WordPress post.
3. We copy and paste the BBC article’s URL to the clipboard
4. We click the CiteIt blockquote button that was added by the CiteIt WordPress plugin
5. We next paste the URL of the article into the prompt, and
6. Click “Publish”
7. Finally, we click the Permalink to view the published results

Because we clicked the “CiteIt blockquote” button, rather than the “CiteIt popup” button, our context displays by expanding the arrows above and below the quote.

B) Now let’s move on to a popup example featuring a quote about Warren Buffett found on Yahoo Finance.

I follow the very same process, copying the text and URL, but this time I select the “CiteIt popup button”.

After I click publish, I don’t see any arrows.
Instead, when I click on the text, I get the popup.

With CiteIt, It’s about as easy to create Contextual Popups as to create links!

From this video, you can see how easy it is to create contextual popups using the Cite.net WordPress plugin.
This plugin is free and open source.

It currently requires the old TinyMCE WordPress editor.
Contact me if you are interested in helping add this functionality to the new Gutenberg editor.

 

Get WordPress plugin

Upgrading Print Archives

What would it take to add contextual citations to the Internet Archive’s print archives?

Citing Document @ The Internet Archive

Take for example a quote from Political Economy (1883)

“Omitting the capital which a joint-stock company puts into a bank, the banker possesses no capital, except his premises ..”

Note the footnote: “Practical Political Economy, 1877, p 452.  (actually page 451 in this edition)

 

Quoted Source

.. which quotes from “Chapters on Practical Political Economy; being the Substance of Lectures delivered in the University of Oxford” (1878):

See quote is drawn from the bottom of page 451

 

Implementation:

Could archive.org use machine learning to:

1. locate the footnotes
2. lookup the source material
3. display the cited material next to the cited page?

Possible User Interface:

Imagine if the footnote were a hyperlink to the print source, of which the internet archive also has.

Inspired by Ted Nelson

Could we make an interface for archive.org’s print items that draws from Ted Nelson‘s ideas?