Music Notation with HTML5 Canvas
I spent a few weekends trying to scratch an itch. While looking for online tools to create and render music notation, I was mildly disappointed. The only real options were crummy flash based sites or clumsy server-side image renderers. Neither of these were particularly appealing.
So I pulled up my sleeves, and cooked up a full-fledged JavaScript API to engrave musical notation directly in the browser with the HTML5 Canvas element.
Here's a demo: HTML5 Music Notation Demo
The library has no external dependencies, and all the glyphs, scores, beams, ties, etc. are positioned and rendered entirely in JavaScript.
One of the cool things about this is that I can now build interactive music sheets that respond to user events like score editors.
So here are all the features I have so far.
- Basic staff and chord support.
- Accidentals and beams.
- Support for all note values with correct rhythmic positioning.
- Support for rests, ties, and multiple voices. (Note the positioning of the notes.)
- Support for guitar tablature.
- Multiple voices in both staff and tab scores.
- Broken bars, odd-timings, custom glyphs, basic interactivity, and a full suite of unit tests (I love QUnit).
I have a fair bit of work to do before I can make it available. In particular, I want to build a simple language that allows users to easily create scores. Something along the lines of the following:
score {
title: Hip Tune
artist: Hip Person
bar { v8 C4 D4 E4 F4 (C4 E4 G4) }
bar { v8 C4 D4 E4 F4 (C4 E4 G4) } repeat 3
}
The above is just an idea - I haven't really given it much thought. But the intent is to make the language easy to read and accessible to non-geeks. I'm open to suggestions related to what the language should look like.
Let me know what you think.
161 comments (imported from blogger)
I had the same idea once and started to build a tab-rendering website, but yours seems to be much nicer, so I am very curious about the resulting code.
Combining this with HTML5 Audio for playing the sheet in real time, combined with a designation seems to be the next logical step. I've looked into it, but it doesn't seem so straightforward as there is no MIDI-support.
In ex. 2, shouldnt the e and the f (with double sharp) on the same beat be pointing in each direction? Like here:
http://imgur.com/QozO7.png
Is there a way to get print resolution higher? Currently in chrome, it looks pixelated when saving as PDF.
Great work!
@Mikkel, yes that's a bug... i'll fix that.
@Anonymous(1) ... you can scale this to any size and have beautiful looking scores, since all the drawing is vector-based (lines and curves).
@BTreeHugger ... no SVG because Canvas has broader browser support... it's more about practicality... (besides, do you think I _like_ coding in Javascript? :-)
@Scott (and others who suggested it)... ABC notation looks like a nice simple language... I'll dig into it some more... (if only it supported tablature...)
as you've seen
), it brilliant to see someone else doing it.
I haven't dug through the code yet to see exactly how you broke down the various problems (defining lengths of bars, making sure bars on related staffs (tab and score in your example) are positioned right, etc.), but I look forward to reading the source as soon as I have a moment free.
Personally, I went the "wrong" way about, basically just drawing immediately to the screen, where I should really have just measured first, and drawn at the end.
As others have commented, the Lilypond site is a great resource.
This essay
was really interesting when I was working on it myself. You've gone well beyond what I did, though, so have probably already seen it.
Also, ABC and lilypond etc exist, but that doesn't necessarily mean they provide the best language in which to write stuff. As soon as you start writing { } in javascript, why not use JSON and maybe a converter to/from the other formats as required?
I second the ABC notation idea also.
Any plans for lyrics/chord notation?
Also release early and often - share the code and we'll help! I actually like javascript :)
http://jashkenas.github.com/coffee-script/
Suggestion to you - your proposed language is so close to JSON - why not just make it JSON and then you can, e.g., just paste it into your Javascript programs?
(Oh, I see someone else has suggested it.... well, do consider it, because JSON is the natural Javascript interchange language...)
Do you have a mailing list for this project?
IRT comments mentioning ABC, MusicXML, lilypond and etc: I think it's useful to keep (and expose) an intermediate representation. You already have an internal representation in Javascript land. You would gain a lot by exposing this interface as JSON.
The rendering engine is really the killer app here. Tying it down to a specific notation, framework, etc is limiting.
I think the best way to approach this would be to open source the rendering engine and the JSON notation format; provide a reference implementation using (say) jQuery and ABC; and let the open-source community develop against their preferred notation, javascript framework, etc.
I think this would provide maximal reusability and benefit to the community.
I would love to work with you on this. This is an itch I've been meaning to scratch for a LONG time. Maybe we can get some hot Github action going? Working on an ABC parser and jQuery wrapper for your rendering engine would be a lot of fun and immediately useful as well.
Cheers!
You should check out what notation they use for music
in LaTeX:
http://icking-music-archive.org/software/indexmt6.html
http://r3jjs.com/js/hdots/
Or you can go to http://code.google.com/p/highlanddots/ and see our latest developmental code.
Looks really neat. We're hacking on HTML5 audio, and together could probably get this playing as well as displaying. Check out our work here http://vocamus.net/dave/?p=1074
Dave
http://www.commodore.ca/manuals/128_system_guide/sect-07b.htm#7.3.2.3
The PLAY statement for the Commodore 128 might help you round out the notation language some.
I agree that a JSON-based API is the way to go, with maybe predefined converters for a common standard like ABC; let folks write their own converters for MusicML or whatever.
Great work so far!
I'd like to second (third?) the notion of changing your implementation to use SVG/VML instead of Canvas. As others have said, Canvas is inaccessible, is raster based (not vector) and JavaScript accessible only. You can use the Raphaël JavaScript library (http://raphaeljs.com) to render the SVG and it will auto-render it in VML for IE6/7/8 (IE9 will support SVG natively).
SVG & VML actually create nodes in the DOM, just like HTML, which you can then attach to and manipulate with libraries like jQuery for endless flexibility. Also makes them visible to screen readers & search engines (assuming you save the generated content to a static resource that can be indexed of course).
I don't see a need to invent a new input format. Use MusicXML for greatest accessibility to the largest number of programs - more than 80 programs can write it out, including Finale, Sibelius, and MuseScore.
If you want easy-to-type input, you can use ABC, but that can be tricky to parse - due both to the nature of the language and to its multiple dialects in common use. You might want to rely on the existing ABC to MusicXML converters instead, and just write one parser.
It's an interesting suggestion above to serialize your internal format, but that doesn't seem to get you a lot of benefit for the time spent. If you read in a standard format, nobody else will have to write a new converter to use this.
Keep up the excellent work!
* You should put the code on GitHub, SourceForge or Google code. The earlier the better.
* Then, as suggested by others, look at Lilypond. That is, for three reasons: a) they have a nice input format, powerful but easy IMHO, (looking quite similar to your initial idea) and b) they feature SVG export, which is interesting, too, when making music for the web. And c) they have an awful rendering engine producing beautiful sheets.
* Talk to the Lilypond team, search for synergy effects and helpers for your code (Open Sourcing, as suggested, helps, too).
Having never properly learned notes myself, but occasionally struggling with them, I love the idea of being able to display notes as Guitar tabs. Please consider adding Ukulele-support :)
I had a need to have a simple music notation that could be quickly typed from a printed score into a web form and then packaged and transmitted within a URL to a music editing service.
Musical objects notated using the small language I developed I call Tunetexts.
Harmony examples can be notated especially quickly using this Tunetext language because notation for chord sequences can be generated by specifying only the chord root and type of chords and duration changes within a chord progression.
A Tunetext server can generate voicings for chords if you don't care to specify them yourself.
If you are doing an example in 4/4 meter, the default duration assumed when a specification starts is assumed to be whole note, so a sequence of whole note chords need only be listed to be performed.
It took about thirty seconds to produce this harmony example, by typing the chord names, specifying the changes of duration that occur and submitting the tunetext URL to the server:
Chords in the key of C Major
The Tunetext Service that will respond to this will return the notation for the score and prepare a MIDI performance of it in a second or two, and present it in a form that provides links that will generate the HTML necessary to present this music on a website in a number of useful ways.
Here is that tunetext URL packaged into a
clickable button
.
Here is a
score animation
generated from the tunetext specification a few seconds after it was submitted to a Score Animation Service that runs on the SongTrellis site.
When it's necessary, the pitches used in a particular chord voicing can be
completely specified
.
Here's a score consisting of
a melody voice and a chord accompaniment
.
I second (or third, or fourth...) the votes for
ABC
compatibility; many of us have libraries of music we've created with ABC markup!
Thanks for sharing!!
John
A Question of Formats
.
ABC notation is nice, although it's true that it doesn't directly support tab. You can always derive tab from standard notation, of course, but the generated fingerings might not be possible or accurate without some work. In particular, I've found the ABCPlus document detailed and easy to read.
Noticed a bug: On iPad, the tab notation is being drawn in the wrong spot. Take a look at
http://imgur.com/AxmbM.png
for a screenshot. If i reload the demo page, the tab notation is drawn correctly, it's only wrong on the first load.
Great job!
http://code.google.com/p/canvas-svg/
Michael
I would like to put an other vote in for the use of MusicXML — as others have stated, it's a widely supported format. There are many applications which can import MIDI, ABC Music, MusiXTeX, LilyPond, ect files and export that into MusicXML... it would be extremely useful to have this done on the server-side (I'm not sure whether there are packages that would be easy to setup already available). Canorus seems to have good libraries: http://freshmeat.net/projects/canorus
Best of luck on this project. Don't let it overwhelm you! I've worked on a JavaMIDI and JavascriptMIDI plugin, let me know whether you would like to plug one of those in (to generate music in real-time).
http://www.musiccog.ohio-state.edu/Humdrum/
that you can easily use to convert to and from other formats.
The notation looks great. Good work!
www.tabnet.fr
No need for MIDI, go for OGG/MP3 samples (just a thought), which are supported by HTML5.
Fantastic idea, by the way!
Best of luck with making it fully functional!
Looking forward to the final version!
This gets still more complicated when voice 2 accidentals "collide" with voice 1 accidentals.
Have you looked at basing this on lilypond scores?
Tim.
might of course be opera's fault though...
This is an idea that I had when I first saw www.noteflight.com.
Interested in making this into an open source collaborative effort?
This is excellent work .. keep it up.
I'm sure this will all come in time. This is a fantastic start and I am very impressed!
Several others have mentioned MIDI. Probably not possible with just JavaScript, but it should be relatively easy to toss all the data to a back end somewhere that would convert it, no?
Fret 2 to 12 or 5 to 23 isn't really feasable.
Remo.D.
Any plans as far as playback? Persisting scores on your server? more?
I'm totally impressed!
Keep up the great work!
L
You might want to look at a book like Gardner Read's or Curt Stone's.
[h2]Test 1[/h2]
[canvas id="notation" width="800" height="400"
style="background: #eed; padding: 10px; border: 10px solid #ddc"]
HTML5 Canvas not supported on this browser.
[/canvas]
Really want to use it to replace lilypond (currently use it with my homegrown marching percussion composition tools). Seems like I'll actually be able to understand your projects code, and thus, contribute!
I've implemented some pretty complex charts for Canvas, but since I've learned about SVG web, I've done everything with SVG, that's the way to go!
Great job! Very interesting to anyone with any music background at all.
Are you considering using a music keyboard for I/O instead of programming notation input to result in a score?
I've always wanted to be able to plug a keyboard into a computer and see the music score appear from what I improvise.
JR
http://www.dimused.uni-tuebingen.de/hildegard/?isForm=1&FILE=1&source=D&VARIANT=1&L=1
This is a rather special case of rendering medieval music with neumes, but uses the open format of the Music Encoding Initiative (MEI) as its data base and shows some of the potentials of SVG's interactivity for rendering variant readings in other sources by mouse over.
Stefan
Abcjs, which is an open source parsing and rendering tool for ABC written
entirely in javascript and allows sheet music to be rendered as both
standard notation and MIDI entirely with the browser. There is a working Demo (with sound) here...
http://drawthedots.com/
Cmin chord: C-4 D#-4 G-4 (where "4" is the octave)
Tempo 4/2 beat:
C-4 D#-4 G-4
. . .
D-4 F-4
. .
C-4 D#-4 G-4
. . .
D-4 F-4
(and so on)
Basically you just need to read it like a spreadsheet, playing one line after another. For faster notes, just add some "off" sign in order to know when the note needs to be stopped:
C-4 C-5
. .
. .
off off
.
D-4 F-4
. .
. .
. off
. .
off .
And so on.
http://youtu.be/9gwwwRBDuYE
You did a great job!
It's a de facto standard extension to the ABC notation language; the abcm2ps typesetter (http://moinejf.free.fr/) also supports tablatures. I'll be glad to give you support if you decide to implement it.
Regards, Guido
Looks great though keep up the GREAT WORK!
A lot of people are mentioning ABC, MusicXML, and LilyPond. You might also want to look at the notation that JFugue uses.
JFugue is a Java-based API for creating music. Its notation was inspired by the Commodore 128's BASIC v7.0. Intended to be easy to learn and use, JFugue focuses on the creation of music as opposed to specific notational elements, but it does closely resemble what you proposed in your sample score script.
Music notation could not be any less suited to XML. Whatever you do, please steer clear of it.
As for musicxml being horrible, I completely agree. But it *is* a more or less established standard. So I think supporting it would be a good idea.
Why do you think about MusicXML this way? In what view is it an abomination?
jTab (http://jtab.tardate.com) is a project I launched last year in a similar vein - although I was particularly focused on guitar tab (and solving the fingering problems that @Octafish mentioned).
At the time, I chose SVG rendering (with Raphael) but have been thinking about a canvas option or port (Raphael/SVG works pretty much everywhere except Google's Android webkit variant!)
http://code.google.com/p/gdbe
http://code.google.com/p/latex-lab/
I just tried to look at your demo and got a 404 message. Are you still working on this? What about making this an open source project?
All the best,
-Joe
I am guessing you have replaced this effort with your open source "VexFlow" project [http://vexflow.com/]. Am I right?
If so you might want to point to that from this posting.
All the best,
Joe
Am thinking about writing conversion to (eventually from) another music notation system in a text format (supports playback & midi export), but haven't written a parsing program in over 17 years.
WBP
For example here is something I made with nted
http://vimeo.com/16894001
Note nted is both the editor and the player for it.
It would be great if this could be done entirely in the browser
I'd like to implement on my music blog here: http://readmusic.org
Many thanks
http://home.comcast.net/~wporter211/realsite/abcnwc.htm
ABC is -except as widely used by folk musicians (and in which it excels) for recalling a tune's melody line- next to useless as an exchange format.
Would it be reasonable to re-work http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=blob;f=lily/lexer.ll;h=2ef59f43f5243618951de8eb6d4c0ea90b23b03f;hb=HEAD using jison or some other JS/CC or something?
Perhaps such an approach would enable a convert-ly to automatically update existing scores?
An important attribute for a such a language is "scribbability," or the ease in setting up a minimal working file. XML fails on scribbability, although it would be fantastic to see something similar to the way the semantic working group developed N3, a scribbable notation for RDF XML.
I'll probably be using this as part of a product (open source hardware) designed to teach people piano.
Cefn
http://cefn.com
html canvas