Building trust in media
This weekend I fixed a bug in the Citeit.net WordPress plugin that prevented authors who have published an article to “Preview” their citations.
This was caused by the “Preview” mode appending querystrings to the permalink, which breaks the quote hash identifier.
This was WordPress plugin bug #11.
On Sunday I finally figured out how to get the CiteIt.net web service working with Docker.
This will allow me to move forward with beta testers.
Yesterday, I tested the latest WordPress plugin with my friend Will. He discovered a validation bug and I entered a new issue in the Github tracker.
There are still quite a few bugs, but I’ve proven that the basic concept is working.
Now that I’m home quarantined for the Coronavirus I have time to document two contributions to CiteIt over the past few months.
I’m hoping to do more work on CiteIt during the Coronavirus quarantine.
Maybe I’ll even find more contributors!
In a previous blog post, I asked “What Bugs Would Bill Gates Find in my Spec?“
If Gates were looking for problems, the first thing he might do is look for Unicode bugs that prevent a computer from matching a citing quote with a source quote.
So, This weekend, I put together a new version of the webservice that I hope reduces the number missed quotes.
The key change involves the hashing mechanism, which previously excluded characters on an ad/hoc basis.
The new system systematically screens all incoming quotes and urls against a list of unicode code points.
The implementation of this is found in the client javascript and server python code.
# Remove the following Unicode code points from Hash
TEXT_ESCAPE_CODE_POINTS = set ( [
2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 39
, 96, 160, 173, 699, 700, 701, 702, 703, 712, 713, 714, 715, 716
, 717, 718, 719, 732, 733, 750, 757, 8211, 8212, 8213, 8216, 8217
, 8219, 8220, 8221, 8226, 8203, 8204, 8205, 65279, 8232, 8233, 133
, 5760, 6158, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200
, 8201, 8202, 8239, 8287, 8288, 12288
])
def escape_text(str):
"""Remove characters from the string"""
str_return = ''
for char in str:
if (ord(char) not in settings.TEXT_ESCAPE_CODE_POINTS):
str_return = str_return + char
return str_return
As a UWaterloo History Grad, I was excited to see a discussion about History on Ontario’s Public Television station.
One quote by UWaterloo’s Ian Mulligan stood out, in which he described the value of studying History:
we try to say come here for critical thinking come here for writing skills come here for your communication skills and that’s fine but that doesn’t really get to the core of what makes history special and the core is to me the core is an understanding of ambiguity and understanding of context the ability to take scattered isolated data points found in an archive or a library here and weave it together into a really compelling story
The other thing that excited me was seeing that Professor Mulligan has an interest in:
Joel Spolsky (best know for cofounding Stack Overflow, FogBugz, Glitch, and Trello) tells the story of his first Bill Gates Review while working as a project manager on the Excel team.
Spolsky recounts that:
Bill doesn’t really want to review your spec, he just wants to make sure you’ve got it under control. His standard M.O. is to ask harder and harder questions until you admit that you don’t know, and then he can yell at you for being unprepared.
For Spolsky’s review, Gates chose to focus on the details of Excel’s date system and its compatibility with Lotus 123. As Spolsky discovered, Basic and Excel use different starting points for the epoch:
.. but Spolsky was puzzled that the two programs yield the same date for today.
I went to find an Excel developer who was old enough to remember why. Ed Fries seemed to know the answer.
“Oh,” he told me. “Check out February 28th, 1900.”
“It’s 59,” I said.
“Now try March 1st.”
“It’s 61!”
“What happened to 60?” Ed asked.
“February 29th. 1900 was a leap year! It’s divisible by 4!”
“Good guess, but no cigar,” Ed said, and left me wondering for a while.
Oops. I did some research. Years that are divisible by 100 are not leap years, unless they’re also divisible by 400.
1900 wasn’t a leap year.
“It’s a bug in Excel!” I exclaimed.
“Well, not really,” said Ed. “We had to do it that way because we need to be able to import Lotus 123 worksheets.”
“So, it’s a bug in Lotus 123?”
“Yeah, but probably an intentional one. Lotus had to fit in 640K. That’s not a lot of memory. If you ignore 1900, you can figure out if a given year is a leap year just by looking to see if the rightmost two bits are zero. That’s really fast and easy. The Lotus guys probably figured it didn’t matter to be wrong for those two months way in the past. It looks like the Basic guys wanted to be anal about those two months, so they moved the epoch one day back.”
Bill Gates, Source: Flickr
When Spolsky met Bill Gates for the meeting, he noticed that Bill had made notes in the margin on every page, suggesting that he had read the whole thing, which impressed Spolsky because Gates had only been given the spec about 24 hours earlier. Gates asked a series of progressively harder and more detailed questions:
Finally the killer question.
“I don’t know, you guys,” Bill said, “Is anyone really looking into all the details of how to do this? Like, all those date and time functions. Excel has so many date and time functions. Is Basic going to have the same functions? Will they all work the same way?”
“Yes,” I said, “except for January and February, 1900.”
Silence.
The f*** counter and my boss exchanged astonished glances. How did I know that? January and February WHAT?
“OK. Well, good work,” said Bill. He took his marked up copy of the spec
Unicode, Source: Wikipedia
If there is an equivalent gotcha issue for CiteIt it concerns the details relating to Unicode escape characters in Javascript and Python.
The problem centers around the way the quote identifier is constructed in the browser in Javascript and on the server in Python. The Python code converts HTML into text, which is a step that exposes spacing and encoding discrepancies when hashed. I discovered these issues through trial and error, but I’ve been wanting to rework my code from scratch.
Yesterday I decided to get serious about improving the escape function which normalizes the text. The current set of functions was put together in an ad-hoc way, but this weekend I decided to do some more thorough research into the Unicode Standard.
Here are some of the things that I found:
It is possible that some non-visible characters could appear in the javascript hash key but not carry over to the python text version when HTML is converted to text, resulting in clashing hashkeys. To avoid that, I plan to remove any characters that may not carry over well when converting from HTML to text.
My new “spec” revision uses a list of escape characters that is 90 characters long.
I was inspired by listening to a 66-minute interview with John O’Nolan about the nonprofit company he started to serve the publishing industry.
When John O’Nolan (@JohnONolan) set out to create Ghost, he made an unintuitive decision for a mission-driven founder: to use his skill set to tackle the obvious thing to work on, rather than chasing the most interesting thing to work on. But 8 years later, and perhaps as a direct result of that decision, Ghost finds itself in one of the most interesting places of any indie business I’ve had on the show: reinventing online publishing in the midst of a crisis for journalism, and making close to $2M/year while doing it.
O’Nolan talks about how the nonprofit structure helps align his incentives with customers and not just look for the big payout when the company sells out to a big player like Google or Facebook.
Tonight I added a new CSP – Content Security Policy to the demo.CiteIt.net website.
The basic idea behind CSPs (Content Security Policies) is that security can be improved if the webserver is configured to tell visitors’ web browsers to only accept files from approved sources. The policy can also specify specific types of actions that can take place, such as whether popups are allowed and whether the site can be accessed over unsecured HTTP (rather than HTTPS).
One of the benefits of applying a CSP to a website is that a public user adding a new sample post would be prevented from embedding a malicious script from a remote source. Or more accurately, web visitors who view a page with an embedded malicious script would be instructed by the webserver not to load this remote script, even if it were successfully injected into the page.
<script src="https://demo.citeit.net/my/script.js" />
<script src="https://evil.attacker.com/bad/script.js" />
Because my CSP defines a set of rules that only allows scripts from self, the web server will instruct the browser not to load the malicious remote script.
These instructions are set by configuring the web server, which in my case is Apache.
Here’s some basic .htaccess code for Apache:
<IfModule mod_headers.c>
Header Content-Security-Policy-Report-Only:
default-src 'self';
script-src 'self' ajax.googleapis.com;
style-src 'self';
img-src 'self';
font-src fonts.googleapis.com, fonts.gstatic.com;
connect-src 'self'; media-src www.youtube.com;
object-src 'none';
prefetch-src read.citeit.net, fonts.googleapis.com, fonts.gstatic.com;
child-src 'self' www.youtube.com;
frame-src 'self' www.youtube.com; worker-src 'none';
frame-ancestors 'self';
form-action 'self';
upgrade-insecure-requests;
block-all-mixed-content;
sandbox allow-forms allow-same-origin allow-scripts allow-popups;
reflected-xss block; base-uri 'self';
manifest-src 'none';
report-uri https://demociteitnet.report-uri.com/r/d/csp/enforce
Header set X-Content-Type-Options nosniff
Header set X-Frame-Options DENY
</IfModule>It is possible to tighten up security further by having the javascript only run a script whose content matches a hash. You see this often with CDNs:
<script src="https://code.jquery.com/jquery-2.1.4.min.js"
integrity="sha384-R4/ztc4ZlRqWjqIuvf6RX5yb/v90qNGx6fS48N0tRxiGkqveZETq72KgDVJCp2TC"
crossorigin="anonymous"></script>
This technique allows the website author to include a remote script from a CDN without worrying that the contents of the script (in this case jQuery) will change. Adding the integrity check prevents the visitor from loading a malicious copy of the script, should the jQuery website get hacked.
Tonight my friend Will installed the CiteIt.net WordPress Plugin on his blog and created a test post. Will is the first person to install the plugin other than myself.
Will had to first install the Classic Editor because the CiteIt.net plugin was designed for WordPress version 4. I have plans for a future version compatible with WordPress Gutenberg (version 5).
Will also discovered some problems in the css style formatting. I’ll update this in a future release.
Last night at the Lancaster Linux meetup, I talked to Alex about improving the Docker Image’s Dockerfile.
I had some trouble with an earlier Python Image (3.6) that didn’t include some of the latest versions of my Python libraries so I switched to a Debian Image with Python 3.7.
He suggested that I used a more recent Alpine Linux Image to save space and to remove some of the extra pip3 calls that created additional layers.
He also suggested setting up docker-compose.
He sent me my first pull request and I integrated it in the Dockerfile.