personnel_sync

package module
v3.2.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 21, 2020 License: MIT Imports: 7 Imported by: 0

README

Personnel Sync

This application is intended to provide a fairly easy to use way of synchronizing people from a personnel system to some other system. In this application there are data sources and destinations. Since destinations have their own unique APIs and integration methods each destination is developed individually to implement the Destination interface. The runtime for this application is configured using a config.json file. An example is provided named config.example.json, however it only has the GoogleGroups destination in it so other supported destinations are documented below.

Sources

REST API

Data sources coming from simple API calls can use the RestAPI source. Here are some examples of how to configure it:

Basic Authentication
{
  "Source": {
    "Type": "RestAPI",
    "ExtraJSON": {
      "Method": "GET",
      "BaseURL": "https://example.com",
      "Path": "/path",
      "ResultsJSONContainer": "Results",
      "AuthType": "basic",
      "Username": "username",
      "Password": "password",
      "CompareAttribute": "email"
    }
  }
}
Bearer Token Authentication
{
  "Source": {
    "Type": "RestAPI",
    "ExtraJSON": {
      "Method": "GET",
      "BaseURL": "https://example.com",
      "Path": "/path",
      "ResultsJSONContainer": "Results",
      "AuthType": "bearer",
      "Password": "token",
      "CompareAttribute": "email"
    }
  }
}
Salesforce OAuth Authentication
{
  "Source": {
    "Type": "RestAPI",
    "ExtraJSON": {
      "Method": "GET",
      "BaseURL": "https://login.salesforce.com/services/oauth2/token",
      "Path": "/services/data/v20.0/query/",
      "ResultsJSONContainer": "records",
      "AuthType": "SalesforceOauth",
      "Username": "admin@example.com",
      "Password": "abc123def.ghiJKL",
      "ClientID": "ABCD1234abcd56789_ABCD1234abcd5678ABCD1234abcd5678ABCD1234abcd5678ABCD1.234abcd5678ABC",
      "ClientSecret": "0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF",
      "CompareAttribute": "email"
    }
  }
}

Destinations

Google Groups

This destination is useful for keeping Google Groups in sync with reports from a personnel system. Below is an example of the destination configuration required for Google Groups:

{
  "Destination": {
    "Type": "GoogleGroups",
    "ExtraJSON": {
      "BatchSize": 10,
      "BatchDelaySeconds": 3,
      "DelegatedAdminEmail": "delegated-admin@domain.com",
      "GoogleAuth": {
        "type": "service_account",
        "project_id": "abc-theme-123456",
        "private_key_id": "abc123",
        "private_key": "-----BEGIN PRIVATE KEY-----\nMIIabc...\nabc...\n...xyz\n-----END PRIVATE KEY-----\n",
        "client_email": "my-sync-bot@abc-theme-123456.iam.gserviceaccount.com",
        "client_id": "123456789012345678901",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-sync-bot%40abc-theme-123456.iam.gserviceaccount.com"
      }
    }
  },
  "AttributeMap": [
    {
      "Source": "First_Name",
      "Destination": "givenName",
      "required": true
    },
    {
      "Source": "Last_Name",
      "Destination": "sn",
      "required": true
    },
    {
      "Source": "Email",
      "Destination": "mail",
      "required": true
    }
  ],
  "SyncSets": [
    {
      "Name": "Sync from personnel to Google Groups",
      "Source": {
          "Path": "/user-report"
      },
      "Destination": {
          "GroupEmail": "group1@groups.domain.com",
          "Owners": ["person_a@domain.com","person_b@domain.com"],
          "Managers": ["another_person@domain.com", "yet-another-person@domain.com"],
          "ExtraOwners": ["google-admin@domain.com"],
          "DisableAdd": false,
          "DisableUpdate": false,
          "DisableDelete": false
      }
    }
  ]
}

Configurations for BatchSize, BatchDelaySeconds, DisableAdd, DisableUpdate, and DisableDelete are all option with defaults as shown in example.

Google Users

This destination can update User records in the Google Directory. The compare attribute is primaryEmail. A limited subset of user properties are available to be updated.

property Google property Google sub-property Google type
id externalIds value organization
area locations area desk
costCenter organizations* costCenter (not set)
department organizations* department (not set)
title organizations* title (not set)
phone phones value work
manager relations value manager
familyName name familyName n/a
givenName name givenName n/a

Custom schema properties can be added using dot notation. For example, a custom property with Field name Building in the custom schema Location is represented as Location.Building.

* CAUTION: updating any field in organizations will overwrite all existing organizations

Following is an example configuration listing all available fields:

{
  "Destination": {
    "Type": "GoogleUsers",
    "ExtraJSON": {
      "BatchSize": 10,
      "BatchDelaySeconds": 3,
      "DelegatedAdminEmail": "admin@example.com",
      "GoogleAuth": {
        "type": "service_account",
        "project_id": "abc-theme-123456",
        "private_key_id": "abc123",
        "private_key": "-----BEGIN PRIVATE KEY-----\nMIIabc...\nabc...\n...xyz\n-----END PRIVATE KEY-----\n",
        "client_email": "my-sync-bot@abc-theme-123456.iam.gserviceaccount.com",
        "client_id": "123456789012345678901",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-sync-bot%40abc-theme-123456.iam.gserviceaccount.com"
      }            
    }
  },
  "AttributeMap": [
    {
      "Source": "email",
      "Destination": "email",
      "required": true
    },
    {
      "Source": "last_name",
      "Destination": "familyName",
      "required": true
    },
    {
      "Source": "first_name",
      "Destination": "givenName",
      "required": true
    },
    {
      "Source": "id",
      "Destination": "id",
      "required": false
    },
    {
      "Source": "phone",
      "Destination": "phone",
      "required": false
    },
    {
      "Source": "area",
      "Destination": "area",
      "required": false
    },
    {
      "Source": "building",
      "Destination": "Location.Building",
      "required": false
    },
    {
      "Source": "cost_center",
      "Destination": "costCenter",
      "required": false
    },
    {
      "Source": "department",
      "Destination": "department",
      "required": false
    },
    {
      "Source": "title",
      "Destination": "title",
      "required": false
    },
    {
      "Source": "manager",
      "Destination": "manager",
      "required": false
    }
  ]
}
Google Service Account Configuration

(see https://stackoverflow.com/questions/53808710/authenticate-to-google-admin-directory-api#answer-53808774 and https://developers.google.com/admin-sdk/reports/v1/guides/delegation)

In the Google Developer Console ...

  • Enable the appropriate API for the Service Account.
  • Create a new Service Account and a corresponding JSON credential file, which should contain something like this:
  {
    "type": "service_account",
    "project_id": "abc-theme-123456",
    "private_key_id": "abc123",
    "private_key": "-----BEGIN PRIVATE KEY-----\nMIIabc...\nabc...\n...xyz\n-----END PRIVATE KEY-----\n",
    "client_email": "my-sync-bot@abc-theme-123456.iam.gserviceaccount.com",
    "client_id": "123456789012345678901",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-sync-bot%40abc-theme-123456.iam.gserviceaccount.com"
  }

These contents will need to be copied into the config.json file as the value of the GoogleAuth key under Destination/ExtraJSON.

In Google Admin Security ...

  • Under "Advanced Settings" add the appropriate API Scopes to the Service Account. Use the numeric client_id.
  • API Scopes required for Google Groups are: https://www.googleapis.com/auth/admin.directory.group and https://www.googleapis.com/auth/admin.directory.group.member
  • The API Scope required for Google User Directory is: https://www.googleapis.com/auth/admin.directory.user

The sync job will need to use the Service Account credentials to impersonate another user that has appropriate domain privileges and who has logged in at least once into G Suite and accepted the terms and conditions. The email address for this user should be stored in the config.json as the DelegatedAdminEmail value under Destination/ExtraJSON.

SolarWinds WebHelpDesk

{
  "AttributeMap": [
      {
        "Source": "FIRST_NAME",
        "Destination": "firstName",
        "required": true,
        "CaseSensitive": true
      },
      {
        "Source": "LAST_NAME",
        "Destination": "lastName",
        "required": true,
        "CaseSensitive": true
      },
      {
        "Source": "EMAIL",
        "Destination": "email",
        "required": true,
        "CaseSensitive": false
      },
      {
        "Source": "USER_NAME",
        "Destination": "username",
        "required": true,
        "CaseSensitive": false
      },
      {
        "Source": "Staff_ID",
        "Destination": "employmentStatus"
      }
  ],
  "Destination": {
    "Type": "WebHelpDesk",
    "ExtraJSON": {
      "URL": "https://whd.mycompany.com/helpdesk/WebObjects/Helpdesk.woa",
      "Username": "syncuser",
      "Password": "apitoken",
      "ListClientsPageLimit": 100,
      "BatchSize": 50,
      "BatchDelaySeconds": 60
    }
  }
}

ListClientsPageLimit, BatchSize and BatchDelaySeconds are optional. Their defaults are as shown in the example config.

Exporting logs from CloudWatch

The log messages in CloudWatch can be viewed on the AWS Management Console. If an exported text or json file is needed, the AWS CLI tool can be used as follows:

aws configure
aws logs get-log-events \
   --log-group-name "/aws/lambda/lambda-name" \
   --log-stream-name '2019/11/14/[$LATEST]0123456789abcdef0123456789abcdef' \
   --output text \
   --query 'events[*].message'

Replace /aws/lambda/lambda-name with the actual log group name and 2019/11/14/[$LATEST]0123456789abcdef0123456789abcdef with the actual log stream. Note the single quotes around the log stream name to prevent the shell from interpreting the $ character. --output text can be changed to --output json if desired. Timestamps are available if needed, but omitted in this example by the --query string.

Documentation

Index

Constants

View Source 
const (
	DefaultConfigFile           = "./config.json"
	DefaultVerbosity            = 5
	DestinationTypeGoogleGroups = "GoogleGroups"
	DestinationTypeGoogleUsers  = "GoogleUsers"
	DestinationTypeWebHelpDesk  = "WebHelpDesk"
	SourceTypeRestAPI           = "RestAPI"
)
View Source 
const (
	VerbosityLow    = 0
	VerbosityMedium = 5
	VerbosityHigh   = 10
)

Variables

This section is empty.

Functions

func GetDesiredAttributes 

func GetDesiredAttributes(attrMap []AttributeMap) []string

func InArray 

func InArray(needle interface{}, haystack interface{}) (exists bool, index int)

This function will search element inside array with any type. Will return boolean and index for matched element. True and index more than 0 if element is exist. needle is element to search, haystack is slice of value to be search.

Types

type AppConfig 

type AppConfig struct {
	Runtime      RuntimeConfig
	Source       SourceConfig
	Destination  DestinationConfig
	AttributeMap []AttributeMap
	SyncSets     []SyncSet
}

func LoadConfig 

func LoadConfig(configFile string) (AppConfig, error)

LoadConfig looks for a config file if one is provided. Otherwise, it looks for a config file based on the CONFIG_PATH env var. If that is not set, it gets the default config file ("./config.json").

type AttributeMap 

type AttributeMap struct {
	Source        string
	Destination   string
	Required      bool
	CaseSensitive bool
}

type BatchTimer 

type BatchTimer struct {
	Counter         int
	SecondsPerBatch int
	BatchSize       int
	// contains filtered or unexported fields
}

BatchTimer is intended as a time limited batch enforcer To create one, call its Init method. Then, to use it call its WaitOnBatch method after every call to 

the associated go routine

func NewBatchTimer 

func NewBatchTimer(batchSize, secondsPerBatch int) BatchTimer

Init sets the startTime to the current time, 

sets the endTime based on secondsPerBatch into the future

func (*BatchTimer) Init 

func (b *BatchTimer) Init(batchSize, secondsPerBatch int)

Init sets the startTime to the current time, 

sets the endTime based on secondsPerBatch into the future

func (*BatchTimer) WaitOnBatch 

func (b *BatchTimer) WaitOnBatch()

WaitOnBatch increments the Counter and then 

if fewer than BatchSize have been dealt with, just returns without doing anything
Otherwise, sleeps until the batch time has expired (i.e. current time is past endTime).
If this last process occurs, then it ends by resetting the batch's times and counter.

type ChangeResults 

type ChangeResults struct {
	Created uint64
	Updated uint64
	Deleted uint64
	Errors  []string
}

func SyncPeople 

func SyncPeople(source Source, destination Destination, config AppConfig) ChangeResults

SyncPeople calls a number of functions to do the following ...

  • it gets the list of people from the source
  • it remaps their attributes to match the keys used in the destination
  • it gets the list of people from the destination
  • it generates the lists of people to change, update and delete
  • if dryRun is true, it prints those lists, but otherwise makes the associated changes

type ChangeSet 

type ChangeSet struct {
	Create []Person
	Update []Person
	Delete []Person
}

func GenerateChangeSet 

func GenerateChangeSet(sourcePeople, destinationPeople []Person, config AppConfig, idField string) ChangeSet

GenerateChangeSet builds the three slice attributes of a ChangeSet (Create, Update and Delete) based on whether they are in the slice 

of destination Person instances.

It skips all source Person instances that have DisableChanges set to true

type Destination 

type Destination interface {
	GetIDField() string
	ForSet(syncSetJson json.RawMessage) error
	ListUsers() ([]Person, error)
	ApplyChangeSet(changes ChangeSet, activityLog chan<- EventLogItem) ChangeResults
}

type DestinationConfig 

type DestinationConfig struct {
	Type          string
	ExtraJSON     json.RawMessage
	DisableAdd    bool
	DisableUpdate bool
	DisableDelete bool
}

type EmptyDestination 

type EmptyDestination struct{}

func (*EmptyDestination) ApplyChangeSet 

func (e *EmptyDestination) ApplyChangeSet(changes ChangeSet, eventLog chan<- EventLogItem) ChangeResults

func (*EmptyDestination) ForSet 

func (e *EmptyDestination) ForSet(syncSetJson json.RawMessage) error

func (*EmptyDestination) GetIDField 

func (e *EmptyDestination) GetIDField() string

func (*EmptyDestination) ListUsers 

func (e *EmptyDestination) ListUsers() ([]Person, error)

type EmptySource 

type EmptySource struct{}

func (*EmptySource) ForSet 

func (e *EmptySource) ForSet(syncSetJson json.RawMessage) error

func (*EmptySource) ListUsers 

func (e *EmptySource) ListUsers(desiredAttrs []string) ([]Person, error)

type EventLogItem 

type EventLogItem struct {
	Event   string
	Message string
}

type Person 

type Person struct {
	CompareValue   string
	ID             string
	Attributes     map[string]string
	DisableChanges bool
}

func RemapToDestinationAttributes 

func RemapToDestinationAttributes(sourcePersons []Person, attributeMap []AttributeMap) ([]Person, error)

RemapToDestinationAttributes returns a slice of Person instances that each have only the desired attributes based on the destination attribute keys. If a required attribute is missing for a Person, then their disableChanges value is set to true.

type RuntimeConfig 

type RuntimeConfig struct {
	DryRunMode bool
	Verbosity  int
}

type Source 

type Source interface {
	ForSet(syncSetJson json.RawMessage) error
	ListUsers(desiredAttrs []string) ([]Person, error)
}

type SourceConfig 

type SourceConfig struct {
	Type      string
	ExtraJSON json.RawMessage
}

type SyncSet 

type SyncSet struct {
	Name        string
	Source      json.RawMessage
	Destination json.RawMessage
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.