The connection properties can be configured using the properties of Managed Connection Factory panel as shown in Figure 1.
Figure 1: Managed Connection Factory panel
Parameters to create the connection can be specified in the input message when this property is set to true. If this property is selected the validation errors in the managed connection factory panel of the CPS are treated as warnings. So user can bypass this step without giving valid configuration and complete the configuration of the component. If valid properties are not provided even in the input message exception will be thrown at runtime.
FTP - File Transfer Protocol is used for transferring files to FTP server.
SFTP - Secure File Transfer Protocol is used for file transfers. If selected, then the property Client Authentication Type is enabled. If Client Authentication Type is set to Password, then client's user name and password are sufficient to login successfully. These details can be set by using the properties Login and Password. If Client Authentication Type is set as Public Key or Both, then the details of Private Key File, Key File Type and Key File Password has to be provided. For detail explanation of SFTP setting, please refer Scenario 3 under section Functional Demonstrations.
Figure 2: SFTP enabled
This property determines the authentication type for the client validation in case of SFTP Protocol.
Figure 3: Client Authentication Type - SFTP
Password: If selected, client's username and password are used for client's authentication.
Public Key: If selected, client's Private Key and the Key File Password are used for client's authentication. (Key file password is different from client password)
Both: This option authenticates using a private/public key-pair, followed by password authentication. If the authentication fails while using client's private key, then it will try to authenticate using password authentication.
The host name/IP address of the machine where the FTP server is running.
The port number on which the FTP server is running.
User name of the FTP user.
Password of the FTP user. This field will be disabled if user selects Protocol as SFTP and Client Authentication Type as Public Key.
The private key file path in the local machine used for client authentication in case of protocol SFTP. The path should include the file name also. The key file should be present on the machine where the peer server (on which peer the component is running) is running. This property is visible when the Protocol is selected as SFTP and Client Authentication Type as Public Key or Both.
Determines the private key type either 'DSA' or 'RSA'. This property is visible when the Protocol is selected as SFTP and Client Authentication Type as Public Key or Both.
The private key file's password when the property protocol is set to SFTP and the property Client Authentication Type is set to Public Key or Both.
Note: Key File password is different from client's password.
Here the user can specify the details for maintaining the pool of connections in the component. On clicking the eclipsis (..) button, Connection Pool Params dialog box appear as shown in the Figure 4.
Figure 4: Connection Pool Params
Enable Connection Pool: If enabled, the connections created are cached in to a pool and used whenever required and available. This can reduce the time for creating a new connection for every input request. If disabled, a new connection is created for each request and it will be closed after completion of that request.
Max Pool Size: The maximum number of connections that can be allocated for the pool.
Blocking Timeout (in ms): The time after which the call to fetch a connection from the pool will timeout if there is no unused connection available.
Idle Timeout (in ms): The time after which the idle connections are returned back to the pool
FTP adapters support HTTP and SOCKS proxies. HTTP is the default option. Here, the user can configure the proxy server settings.
Figure 5: Proxy Settings
Use Proxy Server: If enabled, component will use proxy server settings.
Proxy Address: The IP address or the host name of the machine where the proxy server is running.
Port Number: Port number on which the proxy server is running.
Username: The user name by which the user can login into the proxy server.
Password: Password of the user name by which the user can login into the proxy server.
SOCKS Proxy: Enable this property if the specified proxy server is a SOCKS proxy server.
SSL Security
This property is visible only if the property Protocol is set to FTP.
Figure 6: SSL Settings
Enable SSL: Enable SSL settings to access FTPS (FTP Over SSL) server.
Trust store location: Determines Location of the trust store
Trust store Password: Determines Trust store password
Key store location: Determines Key Store location
Key store Password: Determines Key Store password
Key store Type: Determines Key store type
Trust store Type: Determines Trust store type
Trust Manager Factory Type: Determines Trust Manager Factory type
Key Manager Factory Type: Determines Key Manager Factory type.
Security Provide Class: Determines Security provider class.
Security Protocol: Determines Security protocol
Key Store Client Key: Determines Key Store Client Key
Figure 7: Advanced Settings
The TCP timeout in milliseconds for the sockets. Any operation which takes longer than the timeout value is killed with a java.io.InterruptedException.
The directory on FTP server to which the user's current working directory
will be changed after the login to FTP. The behavior is similar to executing
the command cd <directory_name> after logging in, where <directory_name>
is the value provided for this property.
All relative paths in the server that are computed by the FTPPut component
are relative to this directory.
Example: If the default working directory for the user is /home/user and current directory is set to /home/user/Fiorano, then the working for the user will be changed to /home/user/Fiorano after the user logs into the FTP server.
If the value of property Use Temporary target directory is set to yes and the value for property Temporary target directory is set to temp, then a directory temp will be created under the directory specified by this property.
Determines the type of FTP connection Active or Passive. This property is ignored if Protocol is set as SFTP.
Figure 8: Connect Mode
Active: In Active mode the FTP client specifies the data port that the FTP server is going to connect on and waits for the FTP server to connect. The IP address and port numbers are sent to the FTP server by the FTP client using the PORT command.
Passive: In passive mode the FTP server specifies the data port on which the FTP client connects and waits for the FTP client to connect. The FTP client will ask the FTP server for the server's IP address and port number by issuing the PASV command to the FTP server. This will usually solve the problem of firewalls filtering the incoming data connection.
Specifies the data transfer type
Figure 9: Transfer Type
Ascii: The transferred data is considered to contain only ASCII formatted text. The component is responsible for translating the format of the received text to one that is compatible with the operating system of fps (The fps on which the FTPGet component is running). Text files and files containing HTML, CSS mark-up are suitable for Ascii mode transfer.
Binary: The component transmits raw bytes of the file being transferred. All audio, video and image files are suitable for Binary mode transfer.
Resumes FTP transfer from the point where download has stopped if the transfer is broken. Resume of the broken transfer depends on the FTP server. If the FTP server does not support this then the FTP adapter will start from the beginning otherwise it will start from where it was stopped.
Note: The process of transfer is resume from the broken point only when the Transfer type is Binary.
If files with specific extensions have to be restricted for download from the server, the file extension has to be specified here. This property accepts comma separated list of file extensions. Example: *.zip, *.exe, *.dat.
Example: If this property is set to .exe and the user specifies to put the file named "installer.exe" in the request, then the component ignores that request.
When FTP responses are needed, enabling this property logs all the FTP responses to the Output Log of the component. Figure 10 illustrates a sample snapshot of the debug responses when some download happens.
Figure 10: FTP responses in FTPPuts output log
SITE command parameters
Site commands are sets of extended commands that can be issued by a FTP client, and they are not defined in RFC. However, they are supported by different FTP servers, and different servers usually have different supported site commands. SITE command is used by the server to provide services specific to the system. All the server administrative tasks can be performed by the SITE command.
This property accepts a semicolon separated list of SITE command parameters that have to be executed immediately after login. These parameters are server dependent.
Example: For OS/400 platform, the server specific format of lists or names can be changed to Unix type formats by specifying the value LISTFMT 1; NAMEFMT 1 for this property.
This property is used to parse the directory listing of the FTP Server.
For example, in case of Unix, the directory is listed as follows:
drwxrwxr-x 3 user group 4096 2008-10-23 14:13 fiorano
drwxr-xr-x 14 user group 4096 2008-12-18 14:41 Fiorano
But on Windows, the directory is listed in different format. This listing is the output from the FTP server after executing the dir command.
No: When this property is disabled, the parsing format will be chosen depending on the operating system on which the FTP Server is running.
Yes: Enable this property if a specific parsing format is to be used for parsing the Directory listing returned by the FTP Server.
Example: If a FTP Server is running on IIS on a Windows machine and its directory listing style is set to Unix, enable this property and set 'Parsing format for Directory Listing' to Unix for this property.
This property is used to determine the format to use for parsing directory listing in the FTP Server. The formats supported by the component are Windows and Unix.
Figure 11: Parsing format for Directory Listing
Unix: Select to use Unix format to parse the directory listing
Windows: Select to use Windows format to parse the directory listing
Server connection can be tested from within the CPS by clicking on the Test button in the Managed Connection Factory panel.
Figure 12: Result of successful connection creation
After selecting the appropriate parameters, click the Next button. The Interaction Configurations panel appears as shown in the Figure 12.
Figure 13: Interaction Configurations
Request type
Specifies the type of user input to the adapter. This property provides two options:
File
Data
When the request type is File, the path of the local file which is to be transferred is specified in the input.
Note: Input port appears only when Monitoring is disabled (Value of the property Monitor Directory set to No).
Refer to section Input and Output for details about the effects of these configurations on input and output structures.
When large files are being uploaded to the FTP server, the progress of the transfer can be obtained by specifying yes to this property. If this property is selected as Yes then the FTP adapter will send the notifications of the uploaded process at regular time intervals and this time interval can be specified by the property Monitor Progress interval (in ms). The example output of this notification is shown below. Now, observe the BytesTransferred and TotalBytes fields in the output XML.
Figure 14: Sample output when 'Send Transfer Progress Notification' is set.
If this property is set to true, FTPGet adapter will validate the input request with the input port xsd.
The time interval (in milliseconds) between any two progress notifications. This is visible when the property Send Transfer Progress Notification is set to true.
This closes all the resources except to connection used by the FTPPut adapter after every request. If the less processing time is more important than the less memory usage, then it is recommended to set this property to No and vice versa.
Specifies if the remote file is to be deleted after it is completely uploaded.
Target Namespace for the FTP request and response XML messages.
This property can be used to make the FTPPut adapter poll a directory on the peer servers system(on which the component is launched) for files matching a particular pattern and upload all such files to the server. Enabling this property makes the FTPPut adapter poll the Source directory using the polling configuration specified in Scheduler Configurations panel. The user has to make sure that the Source directory exists.
Note: Properties Source directory, File name patterns, Move to working directory, Working directory, Processed directory, Error directory and Time-based file filtering type are visible only when Monitoring is enabled, that is, the property Monitor Directory is set to yes. Shown in Figure 15 is a sample screen shot with monitoring enabled and the monitoring settings configured. FTPut takes care of the creation of Working directory, Processed directory and Error directory on the system where peer server is running (the peer on which the component is launched), if the directories do not exist Working, Processed and Error directories get created under the Current directory specified in Managed Connection Factory panel. If user does not prefer moves and the creation of these extra directories, then user can set the value for property Move to working directory to NO
Monitoring is done by monitoring the source directory in regular scheduling interval. The scheduler configurations can be defined by the user in Scheduler Configurations Panel and this is the only case that FTPPut component uses scheduler configurations.
Figure 15: Sample monitoring configuration
The directory which contains the files to be uploaded. FTPPut component polls this directory using the polling settings configured in Scheduler Configurations panel.
The type of files in the Source directory which are to be picked up and downloaded. This property accepts multiple file name patterns separated by pipes. Example: *.txt|*.xml|*.exe
When this property is set to yes, the file that has to be uploaded to FTP server will be moved from the directory specified by property Source directory to the directory specified by the property Working directory before the upload begins. The file is read from the working directory and uploaded to the FTP server.
If the upload is successful the file is moved from working directory to the directory specified by property Processed directory.
If the upload is not successful the file is moved from working directory to the directory specified by property Error directory.
When this property is set to no, the file is read directly from the directory specified by property Source directory.
Note: When the user has read-only permission to the file system, this property should be set to no.
This directory holds the files for which the file transfer is in progress.
This directory holds the files for which the upload has been successful.
This directory holds the files for which the upload has failed.
This property provides the capability of monitoring only specific files depending on their modification times. This property provides 4 options (as shown in Figure 16) based on which the files to be monitored could be filtered.
Figure 16: Time-based file filtering types
The behavior is as follows:
NONE: No filtering is applied on the files. Every file present in the Source directory is monitored.
TIME: Files whose last modification time is greater than the last polling time is monitored. This ensures that only the files modified/added after the last polling cycle are monitored.
Figure 17: Time-based file filtering type - TIME
Base Time: Base time in dd:MM:yyyy hh:mm format after which the changed files are to be uploaded
Remote host time offset: If the FTP server and the component are not in the same time zone, the difference in the time zone of FTP Server time zone from the component's time zone should be specified in (+/-) hh:mm format.
HIGHEST_MODIFICATION_TIME: Files whose last modification time is greater than the highest last modification time found in the last polling cycle is monitored. This ensures that only files which are newer than the newer file already polled are selected.
MINIMUM_AGE: Files whose last modification time is less than the current polling time minus the age is monitored. This ensures that the file modification time is at least Minimum Age earlier than the current time.
Figure 18: Time-based file filtering type MINIMUM AGE
Minimum age: The minimum age of the files which are to be monitored.
Note: This property is visible only when the property Time-based file filtering type is set to MINIMUM_AGE.
Remote host time offset: If the FTP server and the component are not in the same time zone, the difference in the time zone of FTP Server time zone from the component's time zone should be specified in (+/-)hh:mm format.
Note: This property is visible only when Time-based file filtering type is set to TIME/HIGHEST_MODIFICATION_TIME/MINIMUM_AGE.
Example: If the source directory contains 4 files named a.txt, b.txt, c.txt and d.txt. The polling interval is 3 min and the first poll is going to start at 11:00:00 (This polling settings can be configure in Scheduler Configuration Panel, please refer the section Scheduler Configuration Panel for more details)
At first poll all files will be monitored irrespective of the value of the property Time-based file filtering type.
If the files have last modification time like this:
a.txt 11:00:16
b.txt 11:00:30
c.txt 11:02:10
d.txt 11:02:50
If the property Time-based file filtering type is set to TIME, then all the files will be monitored in the next poll (which is going to poll at 11:03:00), since all files are modified after the last poll.
If the property Time-based file filtering type is set to HIGHEST_MODIFICATION_TIME, then also all files will be monitored in the next poll (which is going to poll at 11:03:00), since all files are having the last modification time greater than the highest last modification time found on last poll (Component will keep the track of highest last modification time found in the poll).
If the property Time-based file filtering type is set to MINIMUM_AGE and Minimum age is set to 5 min, then no files will be monitored in the next poll (which is going to poll at 11:03:00). Files a.txt and b.txt will be monitored in the polling which will be going to poll at 11:06:00, and the files c.txt and d.txt will be monitored in the polling which will be going to poll at 11:09:00, since the files monitored in the particular polling have been modified at least 5 min ago from the polling time.
Target directory: Directory on the FTP server to which the file(s) is/are to be transferred. Note that this property allows relative paths which would be computed relative to the directory specified for Parent Directory on the local system.
Use Temporary target Directory: If this property is set to true, then the FTPPut adapter will use a temporary target directory for intermediate processing. If you do not prefer to create an extra directory in the FTP server, you can set this property to No.
Temporary target directory: This property is visible only when the property Use temporary target Directory is set to Yes. Directory on the ftp server which the FTPPut component uses for intermediate processing during file downloads.
Note: This directory should not be same as Target directory.
Action that must be taken if the target directory already has a file with name same as the file that is to be uploaded. The behavior will be dependent on the selection as shown below.
Figure 19: Action if same file exists
OVERWRITE: The file to be uploaded overwrites the one in the target directory.
RENAME: The name of the file that is being uploaded is changed based on the properties given below.
Figure 20: Appender counter
Append date-time format postfix: When existing files in the Target directory are not to be overwritten, FTPPut provides the flexibility of uploading the content into a new file whose name is in the format <NameOfExistingFile_CurrentDateTime>. The format in which the date and time is to be appended should be specified as a value for this property.
Example: If the date-time format is specified as MMddyyyyHHmmssss for the file Sample.txt, the target file created would be Sample_0305200811300013.txt.
Append counter: Enabling this property appends a counter along with the Date and Time to target file name, when the target file is not to be overwritten.
Example: A sample file name could be Sample_0305200811300013_0.txt.
APPEND: The file being uploaded gets appended to the one in the target directory.
Note: The same behavior is reflected in the processed directory when Move to working directory is set to Yes.
Auto create temporary directories
The temporary directories created for processing of the component are created based on this property. If this is not set then the temporary directories must be created explicitly.
Note: When the user does not have listing permissions on the server, it is recommended that this property is not set and to create directories manually.
The input and output structures depend the configuration of property Request type.
When Request type is set to File, input and output structures are defined as shown in Figure 21 and Figure 22 respectively.
Figure 21: Input schema structure for the request type - File
Table 1: Input schema element descriptions for - File request type
|
Schema Element |
Description |
|
|
LocalPath |
Path of the local file which is to be uploaded |
|
|
RemoteFile |
File on the FTP server to which the data is to be written |
|
|
Append |
Whether to append data if the file already exists |
|
|
TransferType |
Type of data transfer (ASCII or Binary) |
|
Figure 22: Output schema structure for the request type - File
Table 2: Output schema element descriptions for - File request type
|
Schema Element |
Description |
|
|
LocalPath |
Path of the local file which was transferred to the FTP server |
|
|
RemoteFile |
File on the FTP server to which the data has been written |
|
|
Append |
Append value mentioned in the input |
|
|
TransferType |
TransferType mentioned in the input |
|
|
BytesTransferred |
The number of bytes transferred. |
|
|
TotalBytes |
The total number of bytes transferred |
|
|
ReplyCode |
The reply code sent by the FTP server |
|
|
ReplyText |
The reply text sent by the FTP server |
|
When the type of input is Data, data to be transferred is provided under the Data element in the input message and the data will be written to a file with the name specified under the element RemoteFile.
If text data has to be transferred, then the text content should provided under the Data element in the input message and the attribute dataType should be set to Text.
If binary data has to be transferred, then the base64 encoded string created from binary data should provided under the Data element in the input message and the attribute dataType should be set to Text.
Figure 23 and 24 show the input and output schema structures respectively, when the request type is Data. The only difference in these schema structures against the ones for File request type is the replacement of the schema element LocalPath with Data. Please refer Table 1 and 2 for the remaining schema elements.
Note: Input port appears only when Monitoring is disabled (Value of the property Monitor Directory set to No).
Figure 23: Input schema structure for the request type - Data
Figure 24: Output schema structure for the request type Data
When the property Use Connection details from input is chosen, an additional element ConnectionFactorySettings is added to the input schema, as shown in the figure. Properties that are used to create the connection are present under this element.
Figure 25: Input schema with ConnectionFactorySettings
The configuration can be tested by sending a test file when you click on the Test option in the interaction properties panel.
Figure 26: Sample input sent from CPS
The FTP server can be configured in the connection properties panel of CPS.
Figure 27: Sample output
Send files from a local directory to the FTP servers remote directory.
Configure the FTP Put as described in Configuration and Testing section and use feeder and display components to send sample input and check the response respectively. In the interaction configuration choose the option File for property Request type.
Figure 28: Event Process demonstrating Scenario 1
Sample Input
Figure 29: Input sample when Request type is File.
Sample Output
Figure 30: Output message received for input shown in Figure 29.
Send data to the FTP Server and save it as a file in the remote directory
Configure the FTP Put as described in Configuration and Testing section and use feeder and display component to send sample input and check the response respectively. In the interaction configuration choose the option Data for the property Request type.
Sample Input
Figure 31: Input sample when Request type is Data
Sample Output
Figure 32: Output message received for input shown in Figure 31.
The following steps give a brief description about server settings and SFTP protocol. Here, we provide steps to test FTPPut adapter using vsftpd server which is installed in Linux.
Steps to produce:
Install vsftpd server by executing the following command on command prompt:
yum install vsftpd
Generate keys pairs (both RSA and DSA) and store in ~/.ssh/ directory. For key generation we can use the following commands:
ssh-keygen -t rsa (for RSA type key generation)
ssh-keygen -t dsa (for DSA type key generation)
Note: While executing the above commands, file name and password for the file has to be provided. If the file name provided is id_rsa, then two files will be generated named id_rsa.pub and id_rsa which are public key and private key respectively.
Next, install the public keys in server. Private key has to be with the client which is used to login. The server will authenticate the private key using its public key. This type of client-authentication is called Public Key Authentication. The installation of the public key in the server can be done by the executing following command
ssh-copy-id -i ~/.ssh/id_rsa.pub root@localhost
~/.ssh/id_rsa.pub is rsa public key file path and root@localhost is the server (here we are using the same machine).
Now, change the following in sshd_config file (this file can be located in etc/ssh/ folder). Set RSAAuthentication or DSAAuthentication to yes based on the key file type used.
Add the following line IdentityFile ~/.ssh/id_rsa in the ssh_config file which was located in /etc/ssh. (if already Identity File is set to some other file, then it has to be modified)
Now, restart the servers using the following commands.
/etc/rc.d/init.d/sshd restart
/etc/rc.d/init.d/vsftpd restart
Now the server is ready to accept SFTP protocol to login.
Configuring FTPPut Component
The configuration of the connection properties of FTPPut component for SFTP protocol is shown in the Figure 33, and the remaining procedures are same as explained in Scenario 1.
Figure 33: Protocol type selection
In the revenue control packet example error messages are sent to a FTP server and are stored there for tracking using the FTP Put component.
Figure 34: Demonstration scenario
The event process demonstrating this scenario is bundled with the installer. The bundled process shows it as a File Writer component instead of a FTP Put component.
Documentation of the scenario and instructions to run the flow can be found in the Help tab of flow when open in Studio.
Some FTP Servers have an idle timeout limit for the client connections configured at server side. When a client connection is idle beyond this time limit, server closes the connection. So a new connection has to be made, if a request is made after being idle for time greater than this idle timeout. This means, if the server has a idle timeout value set to 120s and the component does not receive a message at any point of time for more than 120s, the connection would be closed.
There are multiple solutions to this problem -
If the component has to wait for more than 120 seconds before a new message is delivered to the component, then disabling connection pool is a better option.
If the component has to wait for more than 120 seconds in very few cases before a new message is delivered to the component, then enabling reconnection is a better option.
Increase the timeout at server side to a higher value, if possible.
If error with error code 550 occurs during request execution, the component logs the message as resource warning but not as error. To get these messages as errors, the property "Throw fault on warnings" must be selected for request processing error in error configuration panel.