During the past 5 years, my research has centered around grammars:
writing, testing and doing weird things with them. This is an attempt
at build some kind of narrative. There will be 3 parts, and the
purpose of this is just to have words written down in a non-pressuring
format which I can then copypaste into
thesis.tex and make it nicer.
Already before starting my PhD, I was involved in the Grammatical Framework (GF) community, writing and improving grammars in their Resource Grammar Library (RGL). While the grammar writing itself was never the main focus of my PhD, I’ve kept doing it on the side throughout the 5 years.
In 2015, I started writing a Basque grammar–initially it was a joint project with Francis Tyers, but after the first months I took over. Basque is an isolate language with “exotic” features such as absolutive-ergative system and 3-way verb agreement (i.e. verb agrees with subject, object and indirect object), so there was nothing in the RGL to build upon. There was also the additional challenge of me not speaking Basque–below A1 level in any practical things, with a weak grasp of nominal and verbal morphology. I could have some chance of detecting if the morphology was wrong sometimes, but with syntax I had to rely on a grammar book and native speakers.
So how to strategically employ native speakers? You implement a function and show them sentences. This was my workflow in summer 2016, when I had 2 weeks to sit down with a native speaker.
Inari: “Today I’m doing relative clauses, can you help me a bit!”
I can piece together simple sentences, like “the boy drinks beer”, but I start out with no idea whatsoever how to form relative clauses. So I just give the boy and the beer sentence, and ask how to transform that into a relative clause.
Inari: “How do you make this into a relative clause? The boy who drinks beer?”
Informant: “Beer-absolutive drink-relative boy-absolutive.”
Inari: “Okay, and how about The beer that the boy drinks?”
Informant: “Boy-ergative drink-relative beer-absolutive.”
I write first version of the function, and generate 10 sentences, which all use relative clauses. I’m not expecting the function to be perfect, but to prod the informant to come up with more examples: it’s easier to correct an error than to dig the perfect description of Basque relative clauses from your head.
Informant: “What is this sentence supposed to be, our your dog drink beer?”
Inari: “It’s supposed to be our beer, which your dog drinks.”
Informant: “The word order should be: [your dog drink] our beer.”
Inari: “Aha right, I see! And how about these [your dog drink] beers (these beers, which your dog drinks)?”
Informant: “That’s fine.”
Inari: “And these five [your dog drink] beers (these five beers, which your dog drinks)?”
Informant: “That’s wrong too; should be these [your dog drink] five beers; the number five should go at the end like our. But if you have These five yellow beers, then it’s right.”
After some discussion, I add a distinction between “heavy” and “light” modifiers: adjectives are light, relative clauses are heavy. In addition, I split determiners into two classes: numbers and possessives go after heavy modifiers and before light modifiers, the rest go always before all modifiers.
I generate 10 more random sentences which use relative.
Informant: “The relative clauses are correct now. But verb X takes partitive; and if you have partitive, you must put the verb in singular.”
Inari: “Okay, thanks for the info! Quick fix, I’ll force singular agreement with partitive object.”
I generate a new set of examples, which all use partitive.
Informant: “Actually, personal pronouns don’t have partitive; and also, now I remember that negation changes the word order, can you generate some negative sentences?”
Oh but that sounds pretty smooth, so what’s the problem here? It’s
exactly the “generate 10 sentences” part: we’re talking about nothing
more than the
gr command in the GF shell. First I just start off
creating any kind of NP which uses
RelCN, and then I get sentences
like the blue small boy bigger than you whom they won’t have
loved. To combat that, I just type in an example sentence: the beer
which the dog drinks, copy the whole tree, and generate random
sentences with the template
? beer which ? dog drinks. This is how
it looks in the actual tree:
DetCN (DetQuant DefArt NumSg) (RelCN (UseN beer_N) (UseRCl (TTAnt TPres ASimul) PPos (RelSlash IdRP (SlashVP (DetCN (DetQuant DefArt NumSg) (UseN boy_N)) (SlashV2a drink_V2)))))
And I would manually change into question marks the parts where I want variation:
DetCN ? (RelCN (UseN beer_N) (UseRCl (TTAnt TPres ASimul) PPos (RelSlash IdRP (SlashVP (DetCN ? (UseN dog_N)) (SlashV2a drink_V2)))))
This is better than 100 % random the blue small boy bigger than you
whom they won’t have loved type of sentences, but both approaches
miss everything that is formed with a different constructor: for
MassNP (RelCN (UseN beer_N) (UseRCl ...), which uses
MassNP instead of
DetCN. Even in the trees that I do consider, I
need to know what exactly to replace.
Also, my informant happened to be a computational linguist, so it was easy for her to suggest that I generate negative sentences, when I was testing partitive. An ordinary mortal may have easily missed such a detail, and then the test wouldn’t have been as exhaustive.
So how have things improved in 2 years?
We set out to develop an automatic and systematic method for testing grammars. Earlier we relied on the informant to remember and explain all important phenomena to test, and the grammarian to form all relevant trees that show those phenomena. Now, we have a system that takes a GF grammar, a concrete language and a list of functions and/or categories. Then the system generates a representative and minimal set of example sentences that showcases all important variation in the given functions and categories.
Example 1: We test a single word, e.g.
she_Pron in English. The test
program generates contexts that pick out all the different inflection
forms of the word: she, her and hers, and put them into context. For
she_Pron in the category of
Cl returns she is,
there is her right and there is hers. Putting
she_Pron in the
Adv returns more youngly than she, without her, and
without some year of hers.
Example 2: We test a function with arguments, say
PrepNP : Prep -> NP
-> Adv in English and Dutch. Now instead of just having a single word
with already existing fields, we need to generate good combinations of
PrepNP’s arguments, that is,
NPs. The program creates
1 test case for English, e.g.
PrepNP without_Prep (UsePron she_Pron)
without her, and then puts it in a context, there is something
without her. Dutch has more complex phenomena going on, with
prepositions contracting in certain places, so the program needs to
generate 4 test cases to cover all different combinations.
(A note here: if the grammar itself has no semantic coherence, the sentences will be nonsensical as well. Testing an application grammar results in much nicer sentences.)
So, we give the English test case to an English speaker, and the 4
Dutch test cases to a Dutch speaker. If the tester finds one or all of
the sentences grammatically incorrect, we can narrow the bug down to
PrepNP or some of its arguments. For instance, imagine we have
a bug in English, and the output is ✱there is something without she:
she_Pron doesn’t contain the accusative form her, or
doesn’t choose it, but we don’t know which one yet. We can run some
more tests in the program, e.g. test
she_Pron to see all of its forms
in appropriate contexts. When we have an idea where the bug is
located, we fix the bug(s) and then generate the test cases again with
the new grammar.
If all the test cases are grammatically correct, we can be fairly
PrepNP works correctly. Furthermore, we can be
confident that we have seen all relevant distinctions—it would be
redundant to give the 4 Dutch trees linearised into English to the
English speaker, and not enough to give just the one English test case
in Dutch to the Dutch speaker.
To understand the details of how this works, you should read my thesis!
To use the tool in practice, see instructions.