1. Improve readability of your code by following a style guide

2. Avoid sub-queries where possible and create temporary tables, views or use CTEs (common table expressions) like "WITH" (TBH I've never had to use recursive CTE before so you should be fine!)

3. Choose the right primary index column(s) - use columns you will join on or use in a group by and make sure the column has low skew i.e. there is a good distribution of values - to check, simply run `select potential_column, count(*) from table group by 1 order by 2 desc`

4. Know the limitations for temp tables in different systems - e.g. in T-SQL you can collect statistics on temporary tables whilst in Redshift, you need to create the table using DDL then use "into" syntax to retain the original column encodings - learn the intricacies of the system to unlock maximum efficiency

5. Don't trust that the optimiser will automatically do the right thing when joining huge tables - manually filter only the data you need and put them into correctly partitioned temporary tables, then join them!

Knowledge Discovery Overview

Hello All,

I tried to provide an overview of stages involved in knowledge discovery in below video.
This is intended for non-monetary and free knowledge sharing/community mentoring purpose.

https://nusbact.com/2020/02/27/2019-2020-semester-2---session-2/

Thanks to NUS BACT[Business Analytics Consulting Team] for hosting the video on knowledge discovery.

Your feedback is highly appreciated for further improvement/refinement.

Thank you,


Best Regards,
Shriranga Kulkarni.
https://www.linkedin.com/in/shrirangakulkarni/

Knowledge Engineering : Knowledge Management : Knowledge Management(KM) Alignment Table :

In this post, we can see, how Knowledge Management techniques can be strategized to drive the business goals (by aligning with overall company strategy).

Sample Use Case : BreadTalk , Singapore(http://www.breadtalk.com.sg)

                                                Image Source : http://www.breadtalk.com.sg/locations

A staple in the diet of many in Asia, BreadTalk honors the art of bread-making by giving life to their novel creations since its inception. In the continual process of creativity and innovation, Bread Talk’s evolution and growth as an internationally recognized bakery brand over the last 12 years has been encapsulated through three distinctive Generations. And now, BreadTalk is proud to unveil the Generation4 Concept, championed in the campaign 'Get Talking', a global initiative that aims to alter the way customers see, feel and taste the brand.

Their Vision : We have a shared vision to be an international trend-setting lifestyle brand. To this end, we have taken bold strides in introducing new food culture with revolutionary changes and ingenious differentiation. Our products are also crafted with passion and vibrancy to the highest quality. We are confident that our strategies will lend us a distinct competitive advantage and a platform for continued growth.

Key Activities of Knowledge Strategy	Key Objectives of Knowledge Strategy	Key Outcomes of Knowledge Strategy	Strategic Drivers of the Business
Study/Keep Track of Brand Building Trends in Global/Local markets	Identify Key Branding Areas	Develop a plan to implement new Branding Techniques Found (Similar to existing BreadTalk Generation 1 to Generation 4 concepts) and ‘Healthy eating’ concepts.	Building a brand image – Brand Identification
Monitor Recent trends in Franchise markets globally/locally	Identify Key Franchise Setup / Improvement Areas	Maintain Streamlined repository for Franchise Licensing terms and processes depending on locations	Increased and effective Global Presence
Maintain/Manage Customer Information	Improve on Customer  Loyalty	Increase the number of transactions/visits from customers	Increased Profitability
Maintain Customer Feedback System	Analyze  Negatives from Feedback	Improve on negatives(for Customer satisfaction)	Customer Satisfaction
Maintain Customer Feedback System	Analyze  Positives from Feedback	Improve on positives (for Customer Delight)	Customer Delight
Manage Employee Information	Record Employee Plan/Objectives/performance/expectations	Provide Important Perks to Employees/Promotions/ Recreational activities	Employees Satisfaction/Delight
Manage Safety/Security Guidelines	Provide Safety Manuals(for employees) / Display safety guidelines in shops(for customers)	Increased Safety/Security for BreadTalk Ecosystem	Safety-Security of Employee/Customer
Manage Inventory and Supply Information	Build a centralized procurement system	Streamline the Bakery Supply Chain(Daily /Weekly Supplies to all franchises)	Increased Productivity
Manage Training System	Manage (1) Best Practices (2) Core Training (3) Past Experiences	Less dependency on individual employee and streamlined processes.	Decreased dependency on Manpower
Manage Training System and Manage Inventory and Supply Information	Build a centralized procurement system and a training system	Streamline the Bakery Supply Chain(Daily /Weekly Supplies to all franchises) and Increased emphasis on Processes	Overall Effective Cost utilization
Manage IP and Innovations System	Build a centralized IP/Innovation managing system for New concepts/varieties	Study the Positives/negatives of the new BreadTalk Concepts and New Toasts/Products and Apply for Patents	Increased Revenue based on IP/Innovation

Knowledge Engineering : Knowledge Management : Concept Maps

Concept maps are graphical tools for organizing and representing knowledge. They include concepts, usually enclosed in circles or boxes of some type, and relationships between concepts indicated by a connecting line linking two concepts.

A concept map or conceptual diagram is a diagram that depicts suggested relationships between concepts.

It is a graphical tool that instructional designers, engineers, technical writers, and others use to organize and structure knowledge.

A concept map typically represents ideas and information as boxes or circles, which it connects with labeled arrows in a downward-branching hierarchical structure. 

The relationship between concepts can be articulated in linking phrases such requires, or contributes to.

The technique for visualizing these relationships among different concepts is called concept mapping. 

Concept maps define the ontology of computer systems. We can easily say Concept maps can act as effective ways for Knowledge Representation.

There are some popular tools available for concept map drawing, you can refer to below link for more information about concept map drawing tools.

https://ltlatnd.wordpress.com/2011/05/11/ten-popular-concept-mapping-tools/

Following shows a sample concept map on FOOD, on how "Without the industrial chemical reduction of atmospheric nitrogen, starvation would be rampant in developing countries" :

Mule ESB/Java : Reading a CSV/Delimiter file and process line by line

Enterprise Integration : Mule ESB 

Reading Pipe/CSV Files : Process individual lines and display in logger - the each line

Mule Flow Diagram :

Mule Config :

<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:file="http://www.mulesoft.org/schema/mule/file"
xmlns:jdbc="http://www.mulesoft.org/schema/mule/jdbc" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans"  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd">

<configuration doc:name="Configuration">
     <expression-language autoResolveVariables="true">
         <import class="org.mule.util.StringUtils" />
     </expression-language>
</configuration>

    <flow name="ImportCSVFile" >
     <!-- Component 1 : Folder to read CSV file source -->
<file:inbound-endpoint path="C:\InputFileFolderLocation\FileRead" pollingFrequency="5000" doc:name="Source" responseTimeout="10000"/>

<!-- Component 2: Convert between object arrays and strings -->
<object-to-string-transformer doc:name="Object to String"/>

<!-- Component 3: Split each row -->
<splitter expression="#[StringUtils.split(message.payload, '\n\r')]" doc:name="Splitter"/>
        <component class="RequestProcessor1" doc:name="Java"/>    
   
<!-- Component 4:  Paylod - display in logger -->
   <logger message=" #[message.payloadAs(java.lang.String)]" level="INFO" doc:name="Logger"/>
               <!-- Component 5: Transform CSV(",")/Pipe("|") row in array and display in logger -->
              <expression-transformer expression="#[StringUtils.split(message.payload, '|')]" doc:name="Expression"/> 
 
</flow>
   </mule>

Mule Java Component Class and Code:

RequestProcessor1.java(for Java Component):

import java.util.StringTokenizer;

import org.mule.api.MuleEventContext;
import org.mule.api.lifecycle.Callable;

public class RequestProcessor1 implements Callable {

 @Override
public Object onCall(MuleEventContext eventContext) throws Exception {

String[] PayloadStr = eventContext.getMessage().getPayloadAsString().split("\\|", -1);
 Further Processing of PayloadStr ...

}
}...

So, as shown above, we can use the splitter /expression-tranform or splitter/custom java code to read file and process it line by line in Mule ESB.