For those fortunate enough not to know, an FBAR is a form required by the US Government for all US Persons who have assets worth more than $10,000 in another country. And once you have more than that, they want all the details, no matter how big or small. And to make matters worse, while they will allow you to fill in your personal details just once, if you have a joint account, you need to identify that individual for each and every jointly held asset.
And you need to do this every year, even though almost no information changes from year to year.
So, for a long time, I have wanted to automate filling this form in. Normally, I download the PDF version and complete it. Two years ago, I tried to see if I could automate that using the PDFBox tool, but for various reasons that did not work. But last year, I discovered that there is also an online version of the form, and a few weeks ago I discovered the Playwright Chrome Driver library. So …
Let's download Playwright
One of the cool things about Playwright (from my perspective) is that it has an API in Java. So I'm going to use that. In order to build everything, I'm going to use
gradle since that seems quite common these days, and so to start with I'm going to have this
build.gradle file:
plugins {
id 'java'
id 'application'
}
mainClassName = 'ignorance.FBAR'
repositories {
jcenter()
}
dependencies {
implementation 'com.microsoft.playwright:playwright:1.30.0'
}
task copyToLib(type: Copy) {
from configurations.default
into "$buildDir/output/lib"
}
Following along from the
documentation, the first step is to create a central
Playwright instance and then use that to open a browser window. I tend to use Chrome, so that's what I'm doing here, but you can also use Webkit or Firefox.
package ignorance;
import com.microsoft.playwright.Browser;
import com.microsoft.playwright.Page;
import com.microsoft.playwright.Playwright;
class FBAR {
public static void main(String[] argv) {
try (Playwright playwright = Playwright.create()) {
Browser browser = playwright.webkit().launch();
Page page = browser.newPage();
page.navigate("http://whatsmyuseragent.org/");
}
}
}
(In order to get this to work with eclipse, at least for me, I needed to copy the files into a local directory, for which I created the gradle task copyToLib, ran it, and then added the copied JAR files to my classpath).
When I first run this, it downloads a whole bunch of files, which appear to be the browsers it supports. And then, to my (not) very great surprise, it threw an exception:
Caused
by:
com.microsoft.playwright.impl.DriverException:
Error
{
message='Target
closed
name='Error
stack='Error:
Target
closed
Now, I have basically no idea what this means, but I'm going to assume I haven't set something up correctly. Before panicking though, I'm going to try it again. No, still no joy.
Reviewing the code, I realized I was a little over-zealous with my copying and, instead of launching chromium as planned, I launched webkit. I'm not really sure what that does, or what browser it would use, but changing it to chromium certainly solves the problem.
And I also want to see what I am doing, so I have added the headless-off and slowmo options to the launch configuration. And so we have the following:
Browser browser = playwright.chromium().launch(new BrowserType.LaunchOptions().setHeadless(false).setSlowMo(50));
Page page = browser.newPage();
page.navigate("https://bsaefiling1.fincen.treas.gov/lc/content/xfaforms/profiles/htmldefault.html");
Filling in some fields
So, since I don't know what I am doing, I am going to just start randomly filling in the form (my overall plan is to pull all the info from my personal records, probably through a JSON intermediary) using the most stable references I can find. If you haven't pulled the link from the code, the form I'm
filling in is here. And obviously, I have this open in a regular Chrome window with the inspector on so I can find things in it.
It seems to me the best way of identifying the first field - Email Address - is to use the div with class
EmailAddress and then find the input within that. So let's do that and give my official email address -
mickey.mouse@disney.com.
page.fill("div.EmailAddress
input",
"mickey.mouse@disney.com");
OK, that didn't work. It loaded the form, and then paused for a long while (30s to be precise) before giving up and telling me it couldn't find the div:
Timeout
30000ms
exceeded.
===========================
logs
===========================
waiting
for
locator("div.EmailAddress
input")
============================================================
And, after mature reflection, I realized that the div class is actually
Email, not
EmailAddress, so let's try that again:
page.fill("div.Email
input",
"mickey.mouse@disney.com");
Indeed, that does work. So let's quickly fill in the rest of the details in the form and check in.
page.fill("div.Email input", "mickey.mouse@disney.com");
page.fill("div.ConfirmedEmail input", "mickey.mouse@disney.com");
page.fill("div.FirstName input", "Mickey");
page.fill("div.LastName input", "Mouse");
page.fill("div.PhoneNumber input", "770-555-1234");
The next thing to do is to "start" filling in the form. I'm not quite sure why those first fields
don't count as filling in the form, I don't know. This involves pushing the "Start FBAR" button, which is the
click action on the page.
But, as I go to look at the documentation for this, I discover that both
fill and
click on the
Page have been deprecated in favour of
locators, so I am going to digress for a moment into refactoring to use these.
It would seem that this is an attempt to abstract away CSS selectors, and this makes a lot of sense to me. It's more typing, to be sure, but we should never be afraid of trading typing for reliability and correctness.
Interestingly, in doing this, it turns out that there are a number of "duplicate" entries in the form. In particular, because it appears that it just matches
some of the text you provide, the phrase "Enter your email address" also matches the confirmation message. To clarify, it is necessary to add
.setExact(true) to the end. But I have to say, the error message is exceedingly helpful and clear:
Error:
strict
mode
violation:
getByRole(AriaRole.TEXTBOX,
new
Page.GetByRoleOptions().setName("Enter
your
email
address."))
resolved
to
2
elements:
1)
<input
type="text"
class="_O"
name="Email_5"
placehold…/>
aka
getByRole(AriaRole.TEXTBOX,
new
Page.GetByRoleOptions().setName("Enter
your
email
address.").setExact(true))
2)
<input
type="text"
class="_O"
placeholder=""
maxlength…/>
aka
getByRole(AriaRole.TEXTBOX,
new
Page.GetByRoleOptions().setName("Re-enter
your
email
address."))
So, with the refactoring done and the
click() added, let's check in again:
page.getByRole(AriaRole.TEXTBOX, new Page.GetByRoleOptions().setName("Enter your email address.").setExact(true)).fill("mickey.mouse@disney.com");
page.getByRole(AriaRole.TEXTBOX, new Page.GetByRoleOptions().setName("Re-enter your email address.")).fill("mickey.mouse@disney.com");
page.getByRole(AriaRole.TEXTBOX, new Page.GetByRoleOptions().setName("Enter your first name.")).fill("Mickey");
page.getByRole(AriaRole.TEXTBOX, new Page.GetByRoleOptions().setName("Enter your last name.")).fill("Mouse");
page.getByRole(AriaRole.TEXTBOX, new Page.GetByRoleOptions().setName("Enter your telephone number. Do not include formatting such as spaces, dashes, or other punctuation.")).fill("770-555-1234");
page.getByRole(AriaRole.BUTTON, new Page.GetByRoleOptions().setName("Please click this button to begin preparing your FBAR.")).click();
So now we can quickly fill in the next few fields. I am hoping that I will never file this form late again (because it will be so easy when I have this script working!) but in the past few years I have struggled because they moved the deadline from June 30 to April 15 (to align with the US tax year). And I keep forgetting that. So, for the purposes of this blog, I will choose a reason and provide an explanation.
page.getByRole(AriaRole.TEXTBOX,
new
Page.GetByRoleOptions().setName("Filing
name")).fill("Mouse
FBAR
2022");
page.getByRole(AriaRole.COMBOBOX,
new
Page.GetByRoleOptions().setName("reason")).selectOption("A");
page.getByRole(AriaRole.TEXTBOX,
new
Page.GetByRoleOptions().setName("Explanation")).fill("I
keep
forgetting
the
deadline
has
changed.");
Once again, it fails. And once again, playwright's exception message is very clear:
Timeout
30000ms
exceeded.
===========================
logs
===========================
waiting
for
getByRole(AriaRole.TEXTBOX,
new
Page.GetByRoleOptions().setName("Explanation"))
locator
resolved
to
<textarea
class="_k"
placeholder=""
maxlength="750"
tabin…></textarea>
elementHandle.fill("I
keep
forgetting
the
deadline
has
changed.")
waiting
for
element
to
be
visible,
enabled
and
editable
element
is
not
enabled
-
waiting...
============================================================
The element is not enabled. Checking by hand, it seems that "I forgot" is enough of an explanation and that you only need to provide an explanation if you choose "Other". I'm not that bothered, so I'm just going to comment all that lot out and move on.
page.getByRole(AriaRole.TEXTBOX, new Page.GetByRoleOptions().setName("Filing name")).fill("Mouse FBAR 2022");
// page.getByRole(AriaRole.COMBOBOX, new Page.GetByRoleOptions().setName("reason")).selectOption("A");
// page.getByRole(AriaRole.TEXTBOX, new Page.GetByRoleOptions().setName("Explanation")).fill("I keep forgetting the deadline has changed.");
Filling in the joint assets
Right, well, so far, I don't think I've achieved anything very much. As my wife would say, "you could have done
that by hand with a lot less typing". Fair enough. So let's skip to the interesting part of the operation (page 3 is just more information about the primary filer which only needs to be provided once). Parts II and III reflect accounts owned individually or jointly, and the forms can be duplicated by using the appropriate
+ button in the top right hand corner of the page. Now, as noted above, the main thing I want to do is not provide my wife's details ten times on the ten copies of the page for each of the forms (I don't actually want to do
any of it) but this is the thing that drives me crazy).
In order to show the important things, then, I'm going to create two classes now:
Portfolio, which holds all my assets, and
JointAsset which holds the information about a joint asset. Because this shares most of its information with an individually held asset, I'm only going to have this have two fields: an
AccountInfo and a
Asset; I'm going to reuse the former when I go back and fill in Part I.
In the fullness of time, I will extract all the information about the portfolio from its ultimate sources of truth; for now, I am just going to hack something together. Anyway, it's all in
PortfolioLoader.java:
package ignorance;
public class PortfolioLoader {
public Portfolio load() {
Portfolio ret = new Portfolio();
AccountInfo me = new AccountInfo();
AccountInfo other = new AccountInfo();
ret.user(me);
ret.joint(new JointAsset().jointWith(other).setMaximumValue(10000).setType("A"));
return ret;
}
}
All the other classes I created are just boring POJOs, although you could think of them as DTOs between the two systems (the loader and the form-filler).
For now, we are just going to try and load one account. This should not be too difficult. Having said that, we are going to build it as if we are loading multiple accounts and just throw an error if we reach the second.
So, we start by doing the obvious thing:
page.getByRole(AriaRole.TEXTBOX,
new
Page.GetByRoleOptions().setName("*15")).fill(Integer.toString(joint.getMaximumValue()));
which should identify the maximum account value field, but in fact, there are four of them (one in each of sections II, III, IV and V). So that doesn't work that well. We need some means of distinguishing them.
Looking through the structure there is a
div with a class
subForm Part3 and that would seem enough of a distinction. Note that although we would
prefer to use a nice, stable mechanism for identifying the fields, in a pinch it is still possible to use a selector. So let's do that now.
Very good. That works. Let's check in again.
boolean first = true;
for (JointAsset joint : portfolio.joints()) {
if (!first) {
throw new RuntimeException("Not implemented");
}
first = false;
Locator mypage3 = page.locator("div.subform.Part3");
mypage3.getByRole(AriaRole.TEXTBOX, new Locator.GetByRoleOptions().setName("*15")).fill(Integer.toString(joint.getMaximumValue()));
mypage3.getByRole(AriaRole.COMBOBOX, new Locator.GetByRoleOptions().setName("*16")).selectOption(joint.getType());
}
Thread.sleep(10000);
So the one remaining thing I'm interested in experimenting with before I get serious and start integrating things is to try adding a second page for a second asset. Adding the second asset to PortfolioLoader is easy enough:
ret.joint(new JointAsset().jointWith(other).setMaximumValue(10000).setType("A"));
ret.joint(new JointAsset().jointWith(other).setMaximumValue(20000).setType("B"));
return ret;
which is all fine and dandy until we reach the exception we included earlier for the "more than one" case. Now we need to go back and handle that.
The first thing to do is to add another page by clicking on the "+" button. This has "+" as its aria label, so we can do this quite easily:
for (JointAsset joint : portfolio.joints()) {
Locator mypage3 = page.locator("div.subform.Part3");
if (!first) {
mypage3.getByRole(AriaRole.BUTTON, new Locator.GetByRoleOptions().setName("+").setExact(true)).click();
which works first time, but then puts us in the situation where it cannot resolve which of the two "Maximum Value" entries it should be considering:
Error:
strict
mode
violation:
locator("div.subform.Part3").getByRole(AriaRole.TEXTBOX,
new
Locator.GetByRoleOptions().setName("*15"))
resolved
to
2
elements:
1)
<input
class="_s"
value="10000"
type="numeric"
placeho…/>
aka
locator("input[name=\"MaxAcctValue_137\"]")
2)
<input
class="_s"
type="numeric"
placeholder=""
tabind…/>
aka
locator("input[name=\"MaxAcctValueCL_1676554053694\"]")
It turns out that the Locator abstraction is happy to contain one or many potential DOM nodes,
until you decide to do something with them - then it complains about the fact that it cannot choose. But we can easily force that choice using the
last() operator (we could use
first() or
nth(), but since the new forms are added at the end,
last() is what is wanted).
So now we have this code:
Locator mypage3 = page.locator("div.subform.Part3");
if (!first) {
mypage3.getByRole(AriaRole.BUTTON, new Locator.GetByRoleOptions().setName("+").setExact(true)).click();
mypage3 = mypage3.last();
Conclusion
In the space of this blog post, I have convinced myself that Playwright is a reasonable tool for interacting with websites. Hopefully after the conclusion, I will be able to go on and build a tool to import JSON files and fill out FBARs. This will save me a headache for years to come - and may be of use to others too! The repository will be updated to include the final version, even though it is not shown here.
Addendum
In the process of working through the rest of the features, I discovered a number of wrinkles that needed to be resolved.
Selecting the country on this form is a little tricky, as it has an aria-label which is the empty string. I don't think there is anything that can capture that. I solved this problem by instead selecting based on a good, old-fashioned CSS locator. It's not as elegant, but it does the trick.
mypage3.locator("div.partSub div.choicelist.Country select").selectOption(joint.getCountry());
When filling out the address of the owners, it turns out that there is JavaScript logic that connects the country to the list of states. Before a state can be selected, this logic must be run. For whatever reason, this happens in real life but not in Playwright. A little bit of googling suggested that a "blur" event was necessary. (By the way, I discovered the
monitorEvents operation in the Chrome Developer Tools while I was doing this:
check it out).
with.getByRole(AriaRole.COMBOBOX, new Locator.GetByRoleOptions().setName("33")).selectOption(other.getCountry());
with.getByRole(AriaRole.COMBOBOX, new Locator.GetByRoleOptions().setName("33")).dispatchEvent("blur");
Early on, I had tested that I could add additional pages to the document. What I had not considered was that this would add additional
+ buttons. So when I came to add a third asset, it could not tell which
+ button to push. A simple
last() fixed that problem:
mypage3.getByRole(AriaRole.BUTTON, new Locator.GetByRoleOptions().setName("+").setExact(true)).last().click();
At the end of the operation, I manually sign and submit the form - which gives me an opportunity to download what I have submitted. Sadly, this download ended up somewhere in the ether (possibly just in memory) where I could not find it. Before next year, I need to figure this out and save it responsibly to somewhere I can keep records of it.
