You can visit the Troi web site at: www.troi.com

Troi URL Plug-in is copyright 2002-2021 of Troi Automatisering. All rights reserved.

Table of Contents

• Installing plug-ins
• If you have problems
• What can this plug-in do?
• Software requirements
• Getting started
• • Using external functions
  • Where to add the external functions?
  • Using script steps
  • Simple example
  • You can use globals or variables
• How to make sense of a web form
• • Introduction
  • Step 1: Save the HTML source of the page
  • Step 2: Find the <FORM> part in the source
  • Step 3: Remove all formatting, leaving only the fields of the form
  • Step 4: Build the form data to send
  • Using fields as data
  • How to handle radio buttons and checkboxes in a form
  • Pitfalls and considerations
• URL Lengths
• • Use GET and POST in HTML forms - which is better?
• Plug-in limitations and known issues
• Cookies
• Function and script step reference

Installing plug-ins

Starting with FileMaker Pro 12 a plug-in can be installed directly from a container field. Installation is therefore very easy: just open the "All URL Examples.fmp12" example file and the startup script will install the plug-in for you. See also the EasyInstallTroiPlugins.fmp12 example file which contains the actual plug-in and install script.

TIP You can check which plug-ins you have loaded by going to the plug-in preferences: choose Preferences from the Edit menu, and then choose Plug-ins.

You can now open the file "All URL Examples.fmp12" to see how to use the plug-in's functions. There is also a function and script step reference available.

If you have problems

This user guide tries to give you all the information necessary to use this plug-in. So if you have a problem please read this user guide first. You can also visit our support web page:

            www.troi.com/support

This page contains FAQ's (Frequently Asked Questions), help on registration and much more. If that doesn't help you can get free support by email. Send your questions to support@troi.com with a full explanation of the problem. Also give as much relevant information (version of the plug-in, which platform, version of the operating system, version of FileMaker Pro) as possible. Note that due to spam we have to filter incoming email. It might happen that non-spam email is filtered out too. If you have sent an email and you don't get an answer, try to send another email, slightly differently formulated and include the word "FileMaker" in the body text.

If you find any mistakes in this manual or have a suggestion please let us know. We appreciate your feedback!

TIP You can get more information on returned error codes from the OSErrrs database on our web site. This free FileMaker database lists all error codes for Windows and macOS.

What can this plug-in do?

Troi URL Plug-in lets you post and retrieve information from internet. Troi URL Plug-in can help you fill in web forms on internet, all from FileMaker Pro. It also retrieves data from any HTTP URL.
You can:

• use the POST command to fill in a web form and retrieve the result directly in FileMaker
• use the GET command to retrieve data
• use a secure connection (HTTPS) using SSL
• use cookies, custom headers and get data via a proxy server
• easily authorize an account for twitter and post tweets from FileMaker

• fill in forms on web sites or your intranet
• get online data like stock quotes, weather reports, etc.
• automatically add information to listings and online databases
• get images from web sites into FileMaker containers

Software requirements

System requirements for macOS

• macOS 10.14 (Mojave)
• macOS 10.15 (Catalina)
• macOS 11 (Big Sur) including macOS 11.6

NOTE Troi URL Plug-in 8 will probably run on macOS 10.13 (High Sierra), but we have not tested this and no longer provide support for this.

System requirements for Windows

• Windows 8 and 8.1
• Windows 10 including these updates:
  • Creators Update
  • Fall Creators Update
  • April 2018 Update
  • October 2018 Update
  • May 2019 Update
  • November 2019 Update
  • May 2020 Update
  • October 2020 Update
  • May 2021 Update
• Windows 11

FileMaker requirements

FileMaker Pro 16 or FileMaker Pro Advanced 16
FileMaker Pro Advanced 17
FileMaker Pro Advanced 18
FileMaker Pro 19 (including FileMaker Pro 19.3)

FileMaker Server requirements

FileMaker Server 16
FileMaker Server 17
FileMaker Server 18
FileMaker Server 19 (including FileMaker Server 19.3)

You can use FileMaker Server to serve databases that use functions of Troi URL Plug-in (client-side): you need to have the plug-in installed at the clients that use these functions.

Troi URL Plug-in can also be used by FileMaker Server as a server-side plug-in or as a plug-in used by the web publishing engine. To use Troi plug-ins as a server-side or web-side plug-in you need to purchase a special Server/Web license. More information can be found in the download or on our web site.

Getting started

Using external functions

Troi URL Plug-in adds new functions to the standard functions that are available in FileMaker Pro. The functions added by a plug-in are called external functions. You can see those extra functions for all plug-ins at the top right of the Specify Calculation box:

You use special syntax with external functions: FunctionName( parameter1 ; parameter 2) where FunctionName is the name of an external function. A function can have zero or more parameters. Each parameter is separated by a semi-colon. Plug-ins don't work directly after installation. To access a plug-in's function, you need to add the calls to the function in a calculation, for example in a text calculation in Define Fields or in a script in Script Workspace (formaly called ScriptMaker).

Where to add the external functions?

Most of the external functions for this plug-in are intended to be used in a Set Field or Set Variable script step using a calculation. For most functions of this plug-in it makes no sense to add them to a define field calculation, as the functions will have side effects. Only the TURL_ToHTTP function has no side effects and can be used in a define field calculation.

Using script steps

Starting with FileMaker 16 plug-ins can also add script steps. You can select these script steps in the script steps pane in Script Workspace, or you can begin typing the script step name, then choose the script step from the list that appears. The script steps significantly expand the possible actions you can perform with FileMaker Pro.

screenshot of Troi File Plug-in script step

NOTE only steps that perform an action have been added as a script step. Functions that just return information (like for example TURL_IsSecure) are only available as an external function.

Simple example

This example shows how to begin using the Troi URL Plug-in. Say you have a database myGetTest.fmp12, with a global text field called gResult. Create a script "Simple Example". Add the following script step to this script:

  Set Field [ gResult ; TURL_Get ( "-Unused" ; "http://www.example.com/" ) ]

This will get the web page and put it in the field gResult.
The result will be similar to this:

  <HTML><HEAD><TITLE>Example Web Page</TITLE></HEAD>
  <BODY><p>You have reached this web page by typing www.example.com ...These domain
  names are reserved for use in documentation and are not available for registration.
  </p></BODY></HTML>

NOTE Function names, like TURL_Get, are not case sensitive.

Please take a close look at the included example files, as they provide a great starting point. From there you can move on, using the functions of the plug-in as building blocks. Together they give you all the tools you need to interact with internet in powerful ways directly from FileMaker Pro.

You can use globals or variables

With the release of FileMaker Pro 8 and later it is possible to use variables in calculations. Our example files in the download now both use global fields and variables to pass parameters and store the results of a plug-in function.

As this release of Troi URL is intended for FileMaker Pro 16 and higher, we continue to move the scripts to use variables wherever possible. Note that not all examples are using variables yet.

All plug-in functions work with variables just fine. For example if you have these script steps

  Set Field [ this::gURL ; "your URL here" ]
  Set Field [ this::gResult ; TURL_Get ( "-NoDialog" ; this::gURL ) ]

With variables you can alternatively use:

  Set Variable [ $URL ; "your URL here" ]
  Set Variable [ $Result ; TURL_Get ( "-NoDialog" ; $URL ) ]

The main advantage of variables is that you don't need to define global fields that clutter your database definitions. The variables can stay local to the script.

How to make sense of a web form

This section describes how to determine what you must send to make a web form work.

Introduction

On internet there are a lot of web forms. A web form requires one or more fields to be filled before you can use a submit button to send the data to the server. The web server then sends a response. Web forms usually work with the POST method of the HTTP protocol.
It is not that easy to find out the right way to send form data to the server. You need some knowledge of HTML. To find out what to send to a server use the following steps:

1. Save the HTML source of the page
2. Find the <FORM> part in the source
3. Remove all formatting, leaving only the fields of the form
4. Build the form data to send

Step 1: Save the HTML source of the page

In your web browser load the page with the web form on it. Then save the HTML source. On some browsers this can be done with the "Save as" command under the File menu. Choose as format "HTML source" or "Web Page, HTML only" or similar.

Step 2: Find the <FORM> part in the source

Open the file in a Text editor, like BBEdit on Mac or Notepad on Windows. Or use a HTML editor if you have one. The page will look something like the simplified page in figure 1.

  <HTML>
  <HEAD>
    <TITLE>Test Form</TITLE>
  </HEAD>
  <BODY BGCOLOR="#FFFFFF">
  <H2>An example form</H2>
  <P><FORM ACTION="/cgi-sys/formmail.pl" METHOD=POST>
    <P><INPUT TYPE=text NAME=field1 VALUE="" SIZE=30><FONT COLOR="#FF0000"><I>&nbsp;</I></FONT>
    <INPUT TYPE=text NAME=field2 VALUE="" SIZE=30>
    <INPUT TYPE=hidden NAME=secret VALUE=A12474>
    <INPUT TYPE=submit NAME=Submit VALUE="Submit for Processing">
  </FORM></P>
  <P><I>© 2021 Troi, Have Fun!</I></P>
  <P></P>
  </BODY>
  </HTML>

Find the form, by locating the <FORM> tag. The end of the form is indicated with the </FORM> tag. In our example page this is easy, but most web pages are much more complicated.

NOTE there may be more forms on that web page. Make sure you find the correct form.

Remove everything outside the <FORM> </FORM> tags. In our example this will result in figure 2:

  <FORM ACTION="/cgi-sys/formmail.pl" METHOD=POST>
    <P><INPUT TYPE=text NAME=field1 VALUE="" SIZE=30><FONT COLOR="#FF0000"><I>&nbsp;</I></FONT>
    <INPUT TYPE=text NAME=field2 VALUE="" SIZE=30>
    <INPUT TYPE=hidden NAME=secret VALUE=A12474>
    <INPUT TYPE=submit NAME=Submit VALUE="Submit for Processing">
  </FORM>

Step 3: Remove all formatting, leaving only the fields of the form

Remove all formatting tags, and irrelevant text etc. You should leave all the <INPUT ...> tags. This gives:

  <FORM ACTION="/cgi-sys/formmail.pl" METHOD=POST>
  <INPUT TYPE=text NAME=field1 VALUE="" SIZE=30>
  <INPUT TYPE=text NAME=field2 VALUE="" SIZE=30>
  <INPUT TYPE=hidden NAME=secret VALUE=A12474>
  <INPUT TYPE=submit NAME=Submit VALUE="Submit for Processing">
  

Step 4: Build the form data to send

In the ACTION attribute of the FORM tag you can find where the form data should be sent, in this case: "/cgi-sys/ formmail.pl". Assuming that this page is run on the web site www.example.com the total URL would be: "http:// www.example.com/cgi-sys/formmail.pl" Note also the METHOD=POST attribute. This can also be METHOD=GET.

In the form itself you now see there are 3 INPUT fields and a submit field.

field1 with an empty initial value
field2 with an empty initial value
secret with value A12474
Submit with value Submit for Processing

In the POST method you need to concatenate all fields separated with an ampersand "&".

From all this we can infer the data to send. Assuming we want to send data1 as the value for field1 and data2 as the value for field2, then the form data to be sent should be the text:

  field1=data1&field2=data2&secret=A12474&Submit=Submit%20for%20Processing

NOTE Spaces are not allowed, so you need to encode these as %20. Also other higher ASCII characters need to be encoded. You can use the TURL_ToHTTP function of the plug-in to do this.

To finally bring it all together you create a script. In this script you create this script step:

  Set Field [ result ;
    TURL_Post ( "-unused" ;
    "http://www.example.com/cgi-sys/formmail.pl" ;
    "field1=data1&field2=data2&secret=A12474&Submit=Submit%20for%20Processing" ) ]

This will POST the form data to the web server and return the result.

Using fields as data

You can also get the data to send to a server from fields in the database. We assume that you want to send to the same web form as above. In your FileMaker file the following fields should be defined:

The fields field1 and field2 should be filled with the data to send to the server. The field formURL will be set to the URL that handles the form. In Script Workspace add the following script steps:

  Set Field [ formURL ; "http://www.example.com/cgi-sys/formmail.pl" ]
  Set Field [ result ;
    TURL_Post ( "-TimeoutTicks=" & gTimeOutTime ; formURL ;
    "field1=" & field1 & 
    "&field2=" & field2 & 
    "&secret=A12474&Submit=Submit%20for%20Processing" ) ]

This will post the form data to the URL specified and return the result as raw HTML data. If the server does not react, the function will time-out after gTimeOutTime ticks and an error code of "$$-4230" will be returned.

How to handle radio buttons and checkboxes in a form

With the URL plug-in you can also post forms that have radio buttons or checkboxes in it.

Say you want to fill in data on a web page. The web page has this FORM:

  <HTML>
  <HEAD><TITLE>Test Form</TITLE></HEAD>
  <BODY>
  <P>TEST</P>
  <FORM ACTION="/cgi-sys/formmail.pl" METHOD=POST>
    <P><INPUT TYPE=checkbox NAME=checkbox1 VALUE=value1 CHECKED>checklabel</P>
    <P><INPUT TYPE=radio NAME=myradio VALUE=radiovalue1>radiolabel 1
    <INPUT TYPE=radio NAME=myradio VALUE=radiovalue2>radiolabel 2</P>
    <P>input data: <INPUT TYPE=text NAME=field1 VALUE="" SIZE=32><BR>
    <BR>
    <INPUT TYPE=submit NAME=Submit VALUE="Submit for Processing"></P>
  </FORM>
  </BODY>
  </HTML>

This FORM has a checkbox, 2 radio buttons and an input field.

Checkboxes

In the POST data you only include the checkbox if it is checked. Otherwise leave it out. When you add it, add the name, in this case "checkbox1", an equals sign and the value, here "value1".

So in the case above you add: "checkbox1=value1" for this checked box.

Radio buttons

For a radio button you include the value of the radio button that is selected. In this example "myradio", an equals sign and the value of the chosen radio button, here "radiovalue2".

Putting it together you need to send this as the POST data (the 3rd parameter):

  checkbox1=value1&myradio=radiovalue2&field1=typeddata&Submit=Submit%20for%20Processing

Pitfalls and considerations

- Javascripts may move entered data from one field in the web form to another just before the submit is done. Look if this happens in the javascript that's on the web page.

- Servers may use sessionIDs or other ways to make sure the web page is used only once. A sessionID may also time out. This may cause the submission of the web form to fail.

- Web site owners usually don't permit getting data from their web site in any way. Data is usually copyrighted, so be sure to check if you comply with the use of the data.

- Certain special characters and all "high ASCII" characters (characters with an ASCII value above 127) must be encoded when sent to a web server. The function TURL_ToHTTP of the plug-in returns the contents of the specified field or text value encoded in HTTP.

The special characters that must be encoded are
            ; / ? : @ = & > < # % { } ' | \ ^ ~ [ ] ` ̈ © " (space)
plus all high ASCII characters. These characters are encoded using the formula %nn where nn is the hexadecimal value that represents the character in the International Standards Organization (ISO) Latin-1 character set. For example %20 is the encoded value for the space character.

- The submit field may sometimes be left out as not all web servers require it.

URL lengths

The length of URLs may be limited: a lot of web servers don't support URLs longer than 2048 characters. Note that the plug-in has higher limits: the total length of a URL should be less than 32768 characters.

This has consequences for the use of a GET, POST, PUT or DELETE method.

Use GET and POST in HTML forms - which is better?

In HTML, one can specify two different submission methods for a form. The difference between "GET" (the default) and "POST" is primarily defined in terms of form data encoding. The GET method sends extra data as part of the URL, while a POST has a fixed URL and sends the data separately.

If the set-up of the receiving web server is a given, you need to use the method that is specified in the web source. However, if you want to send longer strings of text (more than 2048 characters) you should use the POST method to submit the form.

NB: The official recommendations say that "GET" should be used if and only if the form processing is idempotent, which typically means a pure query form. Generally it is advisable to do so. There are, however, problems related to long URLs and non-ASCII character repertoires which can make it necessary to use "POST" even for idempotent processing.

See http://www.cs.tut.fi/~jkorpela/forms/methods.html for a more in-depth explanation on these issues.

Plug-in limitations and known issues

Troi URL Plug-in 8 still has some limitations. Please be aware of the following:

• The maximum size of a URL is 32768 characters, but see also the section above on URL lengths.
• You can not save data directly to disk. A possible solution is storing the data in a field and then use Troi File Plug-in to write it to disk.
• Error codes on Mac and on Windows can be different in the same situation.
• It is possible to use Troi URL Plug-in to access a file hosted with the Web Companion. However, it won't work always if you are trying to access a URL on the same computer. This is the case if you use the same FileMaker Pro Client for the Web Companion and Troi URL Plug-in.

Cookies

A cookie is a piece of text that a web server can store on a user's hard disk. Cookies allow a web server to store information on a user's machine and later retrieve it. These cookies serve as state information, for example as a unique visitorID.

Cookies are supported by Troi URL Plug-in. On Windows cookies are handled natively by the operating system, meaning that cookies are sent to the web server without the plug-in needing to do something. On macOS you need to use the function TURL_SetCookies to specify which cookie text to send.

Which cookies do you need to send?

Some web servers want to receive cookies before they return the wanted data. In this case you need to find out first which cookies to send. You can do this by returning the headers of a web page in the TURL_Get or TURL_Post functions. For example we get the home page of Amazon:

  Set Field [ result ; TURL_Get ( "-ReturnHeader" ; "http://www.amazon.com/" ) ]

This will result in something similar to this (simplified) page:

  HTTP/1.1 200 OK
    Date: Mon, 27 Dec 2020 16:05:33 GMT
    Server: Stronghold
    Set-Cookie: session-id=104-1127-69752; path=/; domain=.amazon.com; expires=Monday, 07-
    Apr-2021 08:00:00 GMT
    Set-Cookie: session-id-time=1049400; path=/; domain=.amazon.com; expires=Monday, 07-
    Apr-2021 08:00:00 GMT
    Set-Cookie: obidos_path_continue-shopping=continue-shopping-url=/subst/home/home.html/
    104-123397-6952&continue-shopping-post-data=&continue-shopping-
    description=generic.gateway.default; path=/; domain=.amazon.com
    Connection: close
    Content-Type: text/html
    <html> ...
    </hmtl>

Each line that starts with "Set-Cookie:" is a request of the web server to set a cookie on this computer. In this case there are 3 cookies being set. If you want to sent those 3 in the next request add this script step before the next TURL_Get or TURL_Post step:

  Set Field [ gErrorCode ; TURL_SetCookies( "-unused" ;
    "session-id=104-1127-69752; session-id-time=1049400; obidos_path_continue-
    shopping=continue-shopping-url=/subst/home/home.html/104-123397-6952&continue-shopping-
    post-data=&continue-shopping-description=generic.gateway.default
    " ) ]

You need to do this on macOS only. On Windows cookies are handled natively by the operating system. So this function does not do anything on this platform.

See http://www.howstuffworks.com/cookie1.htm for an explanation on how cookies work.

Function and script step reference

For a description of all External Functions and External Script Steps added by Troi URL Plug-in please see the separate Reference.fmp12 file. For each function and script step you will find:

• the correct syntax
• the parameters/options to be used
• the returned result
• special considerations
• an example usage

The same information is also available as online help on Troi's web site.

You can easily consult the online help directly from FileMaker, by clicking the Help button (the small question mark button) next to the function description in the functions pane or the script step description in the script steps pane.