Here’s a new video describing how to use the WordPress plugin:

Get WordPress Plugin

Video Transcript (draft)

Hello, this is Tim Langeman, creator of CiteIt.net.

PROBLEM

1. Today with accusations about Fake News circulating widely, many readers
   suffer under a constant suspicion that any quotation they’ve read may have be taken out of context.
2. Other readers just want to know more about a quote’s context to better understand it’s meaning.

SOLUTION:

I’ve created the CiteIt.net App to address both reader concerns by creating a tool to enable
responsible authors to easily look up and display the contexts of their quotes.

Here is an example of the first type of Contextual Citation: the expanding blockquote:

1. notice the context expands when the reader clicks on an up or down arrow
2. And here’s the second type of contextual citation: the contextual popup which appears when the reader clicks on a quote.

BENEFITS

Authors who use the CiteIt.net App, give their readers more reason to trust them and distinguish themselves from media competitors.

DEMO

You can test-drive the CiteIt.net WordPress plugin by visiting demo.citeit.net

This will send you to the demo site’s WordPress login page.
Log in with the default password and you will be taken to the WordPress dashboard.

To make a test post, click “New -> Post”

Now, let’s create 2 new Contextual Citations from articles I found on Google News:

A) The first article has to with “bats” and is found on the BBC.

To create the citation:

1. We find and highlight a quote.
2. We copy the quote to our clipboard and paste it into to the WordPress post.
3. We copy and paste the BBC article’s URL to the clipboard
4. We click the CiteIt blockquote button that was added by the CiteIt WordPress plugin
5. We next paste the URL of the article into the prompt, and
6. Click “Publish”
7. Finally, we click the Permalink to view the published results

Because we clicked the “CiteIt blockquote” button, rather than the “CiteIt popup” button, our context displays by expanding the arrows above and below the quote.

B) Now let’s move on to a popup example featuring a quote about Warren Buffett found on Yahoo Finance.

I follow the very same process, copying the text and URL, but this time I select the “CiteIt popup button”.

After I click publish, I don’t see any arrows.
Instead, when I click on the text, I get the popup.

With CiteIt, It’s about as easy to create Contextual Popups as to create links!

From this video, you can see how easy it is to create contextual popups using the Cite.net WordPress plugin.

It currently requires the old TinyMCE WordPress editor.
Contact me if you are interested in helping add this functionality to the new Gutenberg editor.

Get WordPress plugin

Upgrading Print Archives

What would it take to add contextual citations to the Internet Archive’s print archives?

Citing Document @ The Internet Archive

Take for example a quote from Political Economy (1883)

“Omitting the capital which a joint-stock company puts into a bank, the banker possesses no capital, except his premises ..”

Note the footnote: “Practical Political Economy, 1877, p 452.  (actually page 451 in this edition)

Political economy by Arthur Latham Perry

Quoted Source

.. which quotes from “Chapters on Practical Political Economy; being the Substance of Lectures delivered in the University of Oxford” (1878):

See quote is drawn from the bottom of page 451

Chapters on Practical Political Economy; being the Substance of Lectures delivered in the University of Oxford by Bonamy Price

Implementation:

Could archive.org use machine learning to:

1. locate the footnotes
2. lookup the source material
3. display the cited material next to the cited page?

Possible User Interface:

Imagine if the footnote were a hyperlink to the print source, of which the internet archive also has.

Political-Economy-Perry Arthur Latham: footnote lookup

Inspired by Ted Nelson

Could we make an interface for archive.org’s print items that draws from Ted Nelson‘s ideas?

You may occasionally notice that the links that you click on deliver you partway down a webpage to a highlighted piece of text.

This is the result of Google Text Fragments —  a feature introduced to Chrome (and available as a Firefox extension) that allows authors to link to a specific piece of text on a webpage.

Link partway down page

Suppose you wanted to quote to a section partway down a page:

Chrome/Firefox Plugin

By using the Chrome/Firefox plugin, an author can construct a link to that specific point of the page.

<a href="https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(idea_lab)#:~:text=Many%20people%20do%20
not%20trust%20quotations%20in%20the%20media%20because%20they%20are%20
suspicious%20that%20the%20quote%20may%20be%20taken%20out%20of%20
context%20or%20fabricated.">
Many people do not trust quotations in the media because they are suspicious that the quote may be taken out of context or fabricated.
</a>

Demo:

Many people do not trust quotations in the media because they are suspicious that the quote may be taken out of context or fabricated.

If you click on the above link in Chrome, it will take you to the point in the page where the selected link text is found and highlight the text:

Fallback Mode

If the reader’s browser does not support text fragments, the reader’s browser will display a normal page, and not scroll down and highlight the term.

Draft Status but Useful Concept?

Even though it is a standard feature on Chrome, Text Fragments is not formally approved by a standards body (W3C draft)  and it requires a plugin for Firefox.  It has. the support of Google, but not all Google projects have staying power.

I’m aware that Brave has raised some privacy concerns and one shouldn’t assume that other browser makers will adopt it.  My main interest is in seeing what do people think of the feature in principle? 

Use in Citation Systems

If copyright issues hamper pulling in greater context, would a feature like Text Fragments be an attractive way to better link the citation with its source?

More Info:

• Ars Technica article
• Chrome plugin
• W3C draft
• Privacy Concern raised by Brave security researcher Peter Snyder

Joe Germuska responded to my question:

How many of the sources journalists quote are not on the web anywhere to be linked?

— Joe Germuska (@JoeGermuska) August 3, 2021

My response:

Non-Web or non-HTML sources are challenges for CiteIt.  There are several approaches for dealing with this:

Granted, It may not always be possible or cost-effective to digitize and publish context for every source.  Sometimes it is more important to be timely than transparent.  If a story is breaking, additional information may be provided at a later time.  A story may also be worth publishing in basic form but not worth the additional investment in contextualizing quote sources.

But if the quote is worth contextualizing, here are some of the options:

Disclaimer: I’ve worked as a newspaper carrier1  but never as a writer or editor, so my thoughts about publishing are uninformed by experience.  That said, here are my thoughts on the options:

1) Minimal Quoting

The quickest, easiest, and least transparent approach to quoting from a non-web source, such as an interview you’ve done or a town council meeting you’ve recorded, is to transcribe only the 500 characters before and after the quote and upload this text as a mini web document.  You can then quote from the document and CiteIt will be able to pull the immediate context.  Not ideal, but better than nothing.

2) Paper -> PDF -> text

If the source is on paper (and typed), you can scan it and upload it. I’ve worked on adding PDF support (code link)  to the web service, but it’s not mature and the OCR support requires more server resources than I can currently support at scale.  In the past, I digitized a copy of War & Peace and it took 16 seconds per page on my average-powered Mac.

It should be possible to add PDF support to the CiteIt webservice if there is enough interest.  (Building out solid support for adding PDF digitization to the web service is in the long-term timeline.)

3) Handwritten Sources

I don’t know how well OCR support handles handwriting.  In these cases, the author may have to revert to the “minimal quoting” method and link to a scan.

4) Citing YouTube Video Transcripts

Many (but not all) YouTube videos have computer-generated transcripts.  The CiteIt webservice currently supports pulling the surrounding context from YouTube videos that contain a transcript.  (Here’s a Sample Malcolm Gladwell YouTube quote that the webservice automatically downloads a transcript for.  The webservice matches the author’s quote against the transcript and retrieves the surrounding text.

5) Self-hosted Audio and Video

Using YouTube to transcribe audio and video is better than not publishing a source at all, but I assume newspapers will want to upload sources to their own site rather than YouTube.

If the newspaper’s media player supports linking to a timestamp, authors can set the starting position and link to the beginning of the quote and let the “reader” choose what parts of the audio they want to listen to.

It would be quite acceptable for the author to transcribe the minimal 500 characters before and after the quote and then let the “reader” examine the remainder of the untranscribed recording.

I’ve also done some work on integrating Google’s Speech to Text for audio and video with CiteIt but I haven’t built out this functionality.  (This is also part of the long-term timeline.)

6) What Types of Quotes are most likely to be Contextualized First:

If CiteIt is used, I imagine it will be featured first and most commonly on articles that are not most time-sensitive.  This means that CiteIt would likely not be used on breaking news but would more likely be used on analysis pieces. Posting the source to YouTube could expedite the transcription of audio and video, but in the long term, software and workflows would have to be developed to streamline the posting of documents and other media on the newspaper’s site.

Your Feedback

I’d be interested in hearing more about this issue from others with experience in the industry.

One of my goals is to “upgrade” all English-language citations to CiteIt’s Contextual Citations.

This will involve:

1. loading all Wikipedia articles
2. finding all footnotes
3. locating the URL corresponding to each footnoted source
4. locating the quote corresponding to each footnote that is delimited with quotation marks.
5. looking up each URL and determining which URL corresponds to the quote
6. wrapping each quote with a <q cite=”URL”>quote</q> tag.

Suggestion: pywikibot #

On the Wikipedia Idea lab site, @Whatamidoing suggested I checkout mw:New Developers site.  There I found the Python  Pywikibot project.

Here’s an example of using Pywikibot (Git Repository) to load a page and perform a replacement:

Wikibase Usage: Get Page #

Wikibase is a flexible knowledge base software that drives Wikidata. A sample pywikibot script for getting data from Wikibase:

import pywikibot
site = pywikibot.Site('wikipedia:en')
repo = site.data_repository()  # the Wikibase repository for given site
page = repo.page_from_repository('Q91')  # create a local page for the given item
item = pywikibot.ItemPage(repo, 'Q91')  # a repository item
data = item.get()  # get all item data from repository for this item

Creating a Custom Page Parsing Bot #

using mwparserfromhell

import pywikibot
from pywikibot import pagegenerators
from pywikibot.bot import ExistingPageBot

class MyBot(ExistingPageBot):

    update_options = {
        'text': 'This is a test text',
        'summary: 'Bot: a bot test edit with Pywikbot.'
    }

    def treat_page(self):
        """Load the given page, do some changes, and save it."""
        text = self.current_page.text
        text += '\n' + self.opt.text
        self.put_current(text, summary=self.opt.summary)

def main():
    """Parse command line arguments and invoke bot."""
    options = {}
    gen_factory = pagegenerators.GeneratorFactory()
    # Option parsing<a name="references"></a>
    local_args = pywikibot.handle_args(args)  # global options
    local_args = gen_factory.handle_args(local_args)  # generators options
    for arg in local_args:
        opt, sep, value = arg.partition(':')
        if opt in ('-summary', '-text'):
            options[opt[1:]] = value
    MyBot(generator=gen_factory.getCombinedGenerator(), **options).run()

if __name == '__main__':
    main()

Iterating over References #

What is the best way to iterate over citation <references>?

Here is what a potential filter would look like:

filter_references(*a, **kw)
"""Iterate over references.<code>

Documentation: Iterating over Tags #

(ReadTheDocs)

This is what existing filters look like:

wikicode Module
class mwparserfromhell.wikicode.Wikicode(nodes)
Bases: mwparserfromhell.string_mixin.StringMixIn
"""A Wikicode is a container for nodes that operates like a string.
Additionally, it contains methods that can be used to extract data from or modify the nodes, implemented in an
interface similar to a list. For example, index() can get the index of a node in the list, and insert() can
add a new node at that index. The filter() series of functions is very useful for extracting and iterating over,
for example, all of the templates in the object."""
RECURSE_OTHERS = 2
append(value)
"""Insert value at the end of the list of nodes.
value can be anything parsable by parse_anything()."""
contains(obj)
"""Return whether this Wikicode object contains obj.
If obj is a Node or Wikicode object, then we search for it exactly among all of our children, recursively.
Otherwise, this method just uses __contains__() on the string."""
filter(*args, **kwargs)
"""Return a list of nodes within our list matching certain conditions.
This is equivalent to calling list() on ifilter()."""
filter_arguments(*a, **kw)
"""Iterate over arguments.
This is equivalent to filter() with forcetype set to Argument."""
filter_comments(*a, **kw)
"""Iterate over comments.
This is equivalent to filter() with forcetype set to Comment."""
filter_external_links(*a, **kw)
"""Iterate over external_links.
This is equivalent to filter() with forcetype set to ExternalLink."""
filter_headings(*a, **kw)
"""Iterate over headings.
This is equivalent to filter() with forcetype set to Heading."""
filter_html_entities(*a, **kw)
"""Iterate over html_entities.
This is equivalent to filter() with forcetype set to HTMLEntity."""
filter_tags(*a, **kw)
"""Iterate over tags.
This is equivalent to filter() with forcetype set to Tag."""
filter_templates(*a, **kw)
"""Iterate over templates.
This is equivalent to filter() with forcetype set to Template."""
filter_text(*a, **kw)
"""Iterate over text.
This is equivalent to filter() with forcetype set to Text."""
filter_wikilinks(*a, **kw)
"""Iterate over wikilinks.
This is equivalent to filter() with forcetype set to Wikilink."""
get(index)
"""Return the indexth node within the list of nodes."""

(Read More)

Video Transcript

Hello, this is Tim Langeman, creator of CiteIt.net.

Often when I read a quotation, I think to myself, “that’s a nice quote, but I wonder what it says two sentences prior, or two sentences after the quote. In other words, what’s the context, and how do I know that this quote wasn’t cherry-picked?

1) Cherry-Picked Quote Example

In Washington DC, the Jefferson Memorial has a badly cherry-picked quote etched into it’s northeastern wall.

Here is the quote: (click quote to view contextual popup window)

Nothing is more certainly written in the book of fate than that these people are to be free.

That sounds great, but it’s just an excerpt.

Using CiteIt.net, authors can link their quotes to their source’s full text, which in this case is  found in the online version of the Yale Law Library:

• http://avalon.law.yale.edu/19th_century/jeffauto.asp

View Quote Context:

The popup window that appears allows the reader to see the following context when they click on the quoted text:

Nor is it less certain that the two races, equally free, cannot live in the same government. Nature, habit, opinion has drawn indelible lines of distinction between them. It is still in our power to direct the process of emancipation and deportation ..

If you look at the context preceding the original quote, you can see Jefferson had argued that slaves “born after a certain date” should be freed and deported “at a proper age”.

CiteIt’s Mission

CiteIt.net’s mission is to develop new digital tools that help combat misinformation and selective quotations.  These tools allow responsible authors to opt into the tool, differentiating themselves from less responsible authors by demonstrating the context surrounding their quotations in order to

• build trust and
• understanding.   

2) Wikipedia as a Stepping Stone:

Though I would like to bring CiteIt.net’s Contextual Citations to Journalism and Academia, I expect these fields will be difficult to enter, especially when CiteIt.net doesn’t yet have the feature set and robustness necessary to support widespread use.

My current strategy is to build CiteIt’s capabilities by converting all the English-language Wikipedia articles to CiteIt-style quotations and use this data set as a testbed and showcase.

A Preview of what’s possible:

So what would Wikipedia look like if it used CiteIt’s type of contextual citations?

You can get a preview by scrolling down the homepage and clicking on 8 sample mockups, such as this manually created Ruth Bader Gingsburg article.

Once you view the Ginsburg article,

1. Scroll down until you see a quote with a grey-blue background, which designates that this quote can be expanded with a popup.
2. Click on the quoted text to see the contextual popup.
3. To read the full article, a reader can click on the “Read More” link in the lower-left corner.

3) More Info

If you are interested in trying out the CiteIt.net WordPress plugin, click on the “Code” tab to view more information.

Web Developers can also add CiteIt-style contextual citations to their sites using sample code available at GitHub.

The “Code” link also includes a “Volunteer” tab which includes information about features I’m looking for help with.

4) Contact

If you have any questions or suggestions, feel free to contact me.

Hello, this is Tim Langeman, creator of CiteIt.net — setting a higher standard of citation, for academics, news organizations, and web authors.

1) The Problem of Out-of-Context Quotations

Often when I read a quotation I think to myself.. “that’s a nice quote but I wonder what it says two sentences prior or two sentences later.” In other words, what’s the context, and how do I know this quote wasn’t cherry-picked?

Jefferson Memorial

In Washington DC, the Jefferson Memorial has a badly cherry-pick quote etched into the northeastern wall. Here it is — “Nothing is more certainly written in the book of fate than that these people are to be free.” That sounds great, but it’s just an excerpt.

Using CiteIt.net, I can link to quote to the full text, making it possible to click on the blue down arrow and reveal the quote’s next 500 characters.

“Nor is it less certain that the two races, equally free, cannot live in the same government. Nature, habit, opinion has drawn indelible lines of distinction between them. It is still in our power to direct the process of emancipation and deportation.

If you look at the text above the quote, you can see that Jefferson was arguing that African Americans should work for a period of time as slaves and then be deported at a “proper age.” Sometimes added context changes everything.

Given how CiteIt.net exposes cherry-picked quotations, no news organization or author attempting to deceive their readers would use it. CiteIt.net isn’t about exposing anyone. Rather, it’s a tool for responsible news organizations and authors who choose to differentiate themselves by holding themselves up to a higher standard, giving readers confidence in their integrity and a greater understanding of their sources.

2) Beyond Quote Verification: Understanding Context

Let’s now look at a second example — this time where the author is not trying to cherry-pick but rather to provide their readers with an easy way to learn more about a quote’s context.

Wikipedia Mockup

I’ve mocked up a list of Wikipedia articles to show what it would look like if Wikipedia were using CiteIt.net’s open-source Contextual Citations.

Here are a few sample articles:

1. Ruth Bader Ginsburg
2. Pride and Prejudice
3. Hamlet
4. Hillary Clinton
5. Donald Trump

I’m looking to bring this feature to all of Wikipedia. Check out the volunteer tab for more information on how you can help

3) Video Citations: Quoting YouTube

Many Americans distrust Press Coverage. For a third example, Let’s take a look at how authors can use the CiteIt.net WordPress plugin to demonstrate their quotations using YouTube video and transcripts.

BBC: Quotation from White House Press Conference

Here’s a sub headline from the BBC about a Trump Administration Press conference. Note the quote is “100%“, without any additional context from the press conference.

US President Donald Trump says he is 100% willing to speak under oath about his conversations with ex-FBI chief James Comey.

• Publishers would always reveal the context of their quotes by linking them to a transcript/video of the original.
• Citizens would be able to easily examine the original event cited by news quotations.
• Politicians would no longer able to excuse their statements by hiding behind the claim that the media took their quotes out of context.