chore: import upstream snapshot with attribution

examples/flows/evaluation/eval-perceived-intelligence/README.md

@@ -0,0 +1,33 @@
+# Perceived Intelligence Evaluation
+
+This is a flow leverage llm to eval perceived intelligence.
+Perceived intelligence is the degree to which a bot can impress the user with its responses, by showing originality, insight, creativity, knowledge, and adaptability.
+
+Tools used in this flow：
+- `python` tool
+- built-in `llm` tool
+
+### 0. Setup connection
+
+Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
+
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base>
+```
+
+### 1. Test flow/node
+
+```bash
+# test with default input value in flow.dag.yaml
+pf flow test --flow .
+```
+
+### 2. create flow run with multi line data
+
+```bash
+pf run create --flow . --data ./data.jsonl --column-mapping question='${data.question}' answer='${data.answer}' context='${data.context}' --stream
+```
+
+You can also skip providing `column-mapping` if provided data has same column name as the flow.
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.

examples/flows/evaluation/eval-perceived-intelligence/aggregate.py

@@ -0,0 +1,21 @@
+from typing import List
+from promptflow.core import tool
+
+
+@tool
+def aggregate(perceived_intelligence_score: List[float]):
+    aggregated_results = {"perceived_intelligence_score": 0.0, "count": 0}
+
+    # Calculate average perceived_intelligence_score
+    for i in range(len(perceived_intelligence_score)):
+        aggregated_results["perceived_intelligence_score"] += perceived_intelligence_score[i]
+        aggregated_results["count"] += 1
+
+    aggregated_results["perceived_intelligence_score"] /= aggregated_results["count"]
+
+    # Log metric for each variant
+    from promptflow.core import log_metric
+
+    log_metric(key="perceived_intelligence_score", value=aggregated_results["perceived_intelligence_score"])
+
+    return aggregated_results

examples/flows/evaluation/eval-perceived-intelligence/flow.dag.yaml

@@ -0,0 +1,262 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+environment:
+  python_requirements_txt: requirements.txt
+inputs:
+  question:
+    type: string
+    default: What is the name of the new language representation model introduced in
+      the document?
+  answer:
+    type: string
+    default: The document mentions multiple language representation models, so it is
+      unclear which one is being referred to as \"new\". Can you provide more
+      specific information or context?
+  context:
+    type: string
+    default: '["statistical language modeling. arXiv preprint arXiv:1312.3005 . Z.
+      Chen, H. Zhang, X. Zhang, and L. Zhao. 2018. Quora question pairs.
+      Christopher Clark and Matt Gardner. 2018. Simple and effective
+      multi-paragraph reading comprehen- sion. In ACL.Kevin Clark, Minh-Thang
+      Luong, Christopher D Man- ning, and Quoc Le. 2018. Semi-supervised se-
+      quence modeling with cross-view training. In Pro- ceedings of the 2018
+      Conference on Empirical Meth- ods in Natural Language Processing , pages
+      1914\u2013 1925. Ronan Collobert and Jason Weston. 2008. A uni\ufb01ed
+      architecture for natural language processing: Deep neural networks with
+      multitask learning. In Pro- ceedings of the 25th international conference
+      on Machine learning , pages 160\u2013167. ACM. Alexis Conneau, Douwe
+      Kiela, Holger Schwenk, Lo \u00a8\u0131c Barrault, and Antoine Bordes.
+      2017. Supervised learning of universal sentence representations from
+      natural language inference data. In Proceedings of the 2017 Conference on
+      Empirical Methods in Nat- ural Language Processing , pages 670\u2013680,
+      Copen- hagen, Denmark. Association for Computational Linguistics. Andrew M
+      Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in
+      neural informa- tion processing systems , pages 3079\u20133087. J. Deng,
+      W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- Fei. 2009. ImageNet: A
+      Large-Scale Hierarchical Image Database. In CVPR09 . William B Dolan and
+      Chris Brockett. 2005. Automati- cally constructing a corpus of sentential
+      paraphrases. InProceedings of the Third International Workshop on
+      Paraphrasing (IWP2005) . William Fedus, Ian Goodfellow, and Andrew M Dai.
+      2018. Maskgan: Better text generation via \ufb01lling in the.arXiv
+      preprint arXiv:1801.07736 . Dan Hendrycks and Kevin Gimpel. 2016. Bridging
+      nonlinearities and stochastic regularizers with gaussian error linear
+      units. CoRR , abs\/1606.08415. Felix Hill, Kyunghyun Cho, and Anna
+      Korhonen. 2016. Learning distributed representations of sentences from
+      unlabelled data. In Proceedings of the 2016 Conference of the North
+      American Chapter of the Association for Computational Linguistics: Human
+      Language Technologies . Association for Computa- tional Linguistics.
+      Jeremy Howard and Sebastian Ruder. 2018. Universal language model
+      \ufb01ne-tuning for text classi\ufb01cation. In ACL. Association for
+      Computational Linguistics. Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng
+      Qiu, Furu Wei, and Ming Zhou. 2018. Reinforced mnemonic reader for machine
+      reading comprehen- sion. In IJCAI . Yacine Jernite, Samuel R. Bowman, and
+      David Son- tag. 2017. Discourse-based objectives for fast un- supervised
+      sentence representation learning. CoRR , abs\/1705.00557.Mandar Joshi,
+      Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large
+      scale distantly supervised challenge dataset for reading comprehen- sion.
+      In ACL. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel,
+      Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought
+      vectors. In Advances in neural information processing systems , pages
+      3294\u20133302. Quoc Le and Tomas Mikolov. 2014. Distributed rep-
+      resentations of sentences and documents. In Inter- national Conference on
+      Machine Learning , pages 1188\u20131196. Hector J Levesque, Ernest Davis,
+      and Leora Morgen- stern. 2011. The winograd schema challenge. In Aaai
+      spring symposium: Logical formalizations of commonsense reasoning , volume
+      46, page 47. Lajanugen Logeswaran and Honglak Lee. 2018. An ef\ufb01cient
+      framework for learning sentence represen- tations. In International
+      Conference on Learning Representations . Bryan McCann, James Bradbury,
+      Caiming Xiong, and Richard Socher. 2017. Learned in translation:
+      Con-","tool for measuring readability. Journalism Bulletin ,
+      30(4):415\u2013433. Erik F Tjong Kim Sang and Fien De Meulder. 2003.
+      Introduction to the conll-2003 shared task: Language-independent named
+      entity recognition. In CoNLL . Joseph Turian, Lev Ratinov, and Yoshua
+      Bengio. 2010. Word representations: A simple and general method for
+      semi-supervised learning. In Proceedings of the 48th Annual Meeting of the
+      Association for Compu- tational Linguistics , ACL \u201910, pages
+      384\u2013394. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
+      Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.
+      Attention is all you need. In Advances in Neural Information Pro- cessing
+      Systems , pages 6000\u20136010. Pascal Vincent, Hugo Larochelle, Yoshua
+      Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust
+      features with denoising autoen- coders. In Proceedings of the 25th
+      international conference on Machine learning , pages 1096\u20131103. ACM.
+      Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and
+      Samuel Bowman. 2018a. Glue: A multi-task benchmark and analysis
+      platformfor natural language understanding. In Proceedings of the 2018
+      EMNLP Workshop BlackboxNLP: An- alyzing and Interpreting Neural Networks
+      for NLP , pages 353\u2013355. Wei Wang, Ming Yan, and Chen Wu. 2018b.
+      Multi- granularity hierarchical attention fusion networks for reading
+      comprehension and question answering. InProceedings of the 56th Annual
+      Meeting of the As- sociation for Computational Linguistics (Volume 1: Long
+      Papers) . Association for Computational Lin- guistics. Alex Warstadt,
+      Amanpreet Singh, and Samuel R Bow- man. 2018. Neural network acceptability
+      judg- ments. arXiv preprint arXiv:1805.12471 . Adina Williams, Nikita
+      Nangia, and Samuel R Bow- man. 2018. A broad-coverage challenge corpus for
+      sentence understanding through inference. In NAACL . Yonghui Wu, Mike
+      Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey,
+      Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.
+      Google\u2019s neural ma- chine translation system: Bridging the gap
+      between human and machine translation. arXiv preprint arXiv:1609.08144 .
+      Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How
+      transferable are features in deep neural networks? In Advances in neural
+      information processing systems , pages 3320\u20133328. Adams Wei Yu, David
+      Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V
+      Le. 2018. QANet: Combining local convolution with global self-attention
+      for reading comprehen- sion. In ICLR . Rowan Zellers, Yonatan Bisk, Roy
+      Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset
+      for grounded commonsense inference. In Proceed- ings of the 2018
+      Conference on Empirical Methods in Natural Language Processing (EMNLP) .
+      Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun,
+      Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies:
+      Towards story-like visual explanations by watching movies and reading
+      books. In Proceedings of the IEEE international conference on computer
+      vision , pages 19\u201327. Appendix for \u201cBERT: Pre-training of Deep
+      Bidirectional Transformers for Language Understanding\u201d We organize
+      the appendix into three sections: \u2022 Additional implementation details
+      for BERT are presented in Appendix A;\u2022 Additional details for our
+      experiments are presented in Appendix B; and \u2022 Additional ablation
+      studies are presented in Appendix C. We present additional ablation
+      studies for BERT including: \u2013Effect of Number of Training Steps; and
+      \u2013Ablation for Different"]} {"question": "What is the main difference
+      between BERT and previous language representation models?", "variant_id":
+      "v1", "line_number": 2, answer":"BERT is designed to pre-train deep
+      bidirectional representations from unlabeled text by jointly conditioning
+      on both left and right context in all layers, allowing it to incorporate
+      context from both directions. This is unlike previous language
+      representation models that are unidirectional, which limits the choice of
+      architectures that can be used during pre-training and could be
+      sub-optimal for sentence-level tasks and token-level tasks such as
+      question answering.","context":["BERT: Pre-training of Deep Bidirectional
+      Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton
+      Lee Kristina Toutanova Google AI Language
+      fjacobdevlin,mingweichang,kentonl,kristout g@google.com Abstract We
+      introduce a new language representa- tion model called BERT , which stands
+      for Bidirectional Encoder Representations from Transformers. Unlike recent
+      language repre- sentation models (Peters et al., 2018a; Rad- ford et al.,
+      2018), BERT is designed to pre- train deep bidirectional representations
+      from unlabeled text by jointly conditioning on both left and right context
+      in all layers. As a re- sult, the pre-trained BERT model can be \ufb01ne-
+      tuned with just one additional output layer to create state-of-the-art
+      models for a wide range of tasks, such as question answering and language
+      inference, without substantial task- speci\ufb01c architecture
+      modi\ufb01cations. BERT is conceptually simple and empirically powerful.
+      It obtains new state-of-the-art re- sults on eleven natural language
+      processing tasks, including pushing the GLUE score to 80.5% (7.7% point
+      absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute
+      improvement), SQuAD v1.1 question answer- ing Test F1 to 93.2 (1.5 point
+      absolute im- provement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute
+      improvement). 1 Introduction Language model pre-training has been shown to
+      be effective for improving many natural language processing tasks (Dai and
+      Le, 2015; Peters et al., 2018a; Radford et al., 2018; Howard and Ruder,
+      2018). These include sentence-level tasks such as natural language
+      inference (Bowman et al., 2015; Williams et al., 2018) and paraphrasing
+      (Dolan and Brockett, 2005), which aim to predict the re- lationships
+      between sentences by analyzing them holistically, as well as token-level
+      tasks such as named entity recognition and question answering, where
+      models are required to produce \ufb01ne-grained output at the token level
+      (Tjong Kim Sang and De Meulder, 2003; Rajpurkar et al., 2016).There are
+      two existing strategies for apply- ing pre-trained language
+      representations to down- stream tasks: feature-based and\ufb01ne-tuning .
+      The feature-based approach, such as ELMo (Peters et al., 2018a), uses
+      task-speci\ufb01c architectures that include the pre-trained
+      representations as addi- tional features. The \ufb01ne-tuning approach,
+      such as the Generative Pre-trained Transformer (OpenAI GPT) (Radford et
+      al., 2018), introduces minimal task-speci\ufb01c parameters, and is
+      trained on the downstream tasks by simply \ufb01ne-tuning allpre- trained
+      parameters. The two approaches share the same objective function during
+      pre-training, where they use unidirectional language models to learn
+      general language representations. We argue that current techniques
+      restrict the power of the pre-trained representations, espe- cially for
+      the \ufb01ne-tuning approaches. The ma- jor limitation is that standard
+      language models are unidirectional, and this limits the choice of archi-
+      tectures that can be used during pre-training. For example, in OpenAI GPT,
+      the authors use a left-to- right architecture, where every token can only
+      at- tend to previous tokens in the self-attention layers of the
+      Transformer (Vaswani et al., 2017). Such re- strictions are sub-optimal
+      for sentence-level tasks, and could be very harmful when applying
+      \ufb01ne- tuning based approaches to token-level tasks such as question
+      answering, where it is crucial to incor- porate context from both
+      directions. In this paper, we improve the \ufb01ne-tuning based approaches
+      by proposing BERT: Bidirectional Encoder Representations from
+      Transformers.","the self-attention layers of the Transformer (Vaswani et
+      al., 2017). Such re- strictions are sub-optimal for sentence-level tasks,
+      and could be very harmful when applying \ufb01ne- tuning based approaches
+      to token-level tasks such as question answering, where it is crucial to
+      incor- porate context from both directions. In this paper, we improve the
+      \ufb01ne-tuning based approaches by proposing BERT: Bidirectional Encoder
+      Representations from Transformers. BERT alleviates the previously
+      mentioned unidi- rectionality constraint by using a \u201cmasked lan-
+      guage model\u201d (MLM) pre-training objective, in- spired by the Cloze
+      task (Taylor, 1953). The masked language model randomly masks some of the
+      tokens from the input, and the objective is to predict the original
+      vocabulary id of the maskedarXiv:1810.04805v2 [cs.CL] 24 May 2019word
+      based only on its context. Unlike left-to- right language model
+      pre-training, the MLM ob- jective enables the representation to fuse the
+      left and the right context, which allows us to pre- train a deep
+      bidirectional Transformer. In addi- tion to the masked language model, we
+      also use a \u201cnext sentence prediction\u201d task that jointly pre-
+      trains text-pair representations. The contributions of our paper are as
+      follows: \u2022 We demonstrate the importance of bidirectional
+      pre-training for language representations. Un- like Radford et al. (2018),
+      which uses unidirec- tional language models for pre-training, BERT uses
+      masked language models to enable pre- trained deep bidirectional
+      representations. This is also in contrast to Peters et al. (2018a), which
+      uses a shallow concatenation of independently trained left-to-right and
+      right-to-left LMs. \u2022 We show that pre-trained representations reduce
+      the need for many heavily-engineered task- speci\ufb01c architectures.
+      BERT is the \ufb01rst \ufb01ne- tuning based representation model that
+      achieves state-of-the-art performance on a large suite of sentence-level
+      andtoken-level tasks, outper- forming many task-speci\ufb01c
+      architectures. \u2022 BERT advances the state of the art for eleven NLP
+      tasks. The code and pre-trained mod- els are available at
+      https:\/\/github.com\/ google-research\/bert . 2 Related Work There is a
+      long history of pre-training general lan- guage representations, and we
+      brie\ufb02y review the most widely-used approaches in this section. 2.1
+      Unsupervised Feature-based Approaches Learning widely applicable
+      representations of words has been an active area of research for decades,
+      including non-neural (Brown et al., 1992; Ando and Zhang, 2005; Blitzer et
+      al., 2006) and neural (Mikolov et al., 2013; Pennington et al., 2014)
+      methods. Pre-trained word embeddings are an integral part of modern NLP
+      systems, of- fering signi\ufb01cant improvements over embeddings learned
+      from scratch (Turian et al., 2010). To pre- train word embedding vectors,
+      left-to-right lan- guage modeling objectives have been used (Mnih and
+      Hinton, 2009), as well as objectives to dis- criminate correct from
+      incorrect words in left and right context (Mikolov et al., 2013).These
+      approaches have been generalized to coarser granularities, such as
+      sentence embed- dings (Kiros et al., 2015; Logeswaran and Lee, 2018). "]'
+outputs:
+  perceived_intelligence_score:
+    type: string
+    reference: ${parse_score.output}
+nodes:
+- name: parse_score
+  type: python
+  source:
+    type: code
+    path: parse_score.py
+  inputs:
+    gpt_score: ${gpt_perceived_intelligence.output}
+- name: aggregate
+  type: python
+  source:
+    type: code
+    path: aggregate.py
+  inputs:
+    perceived_intelligence_score: ${parse_score.output}
+  aggregation: true
+- name: gpt_perceived_intelligence
+  type: llm
+  source:
+    type: code
+    path: gpt_perceived_intelligence.md
+  inputs:
+    # This is to easily switch between openai and azure openai.
+    # deployment_name is required by azure openai, model is required by openai.
+    deployment_name: gpt-4
+    model: gpt-4
+    max_tokens: 5
+    answer: ${inputs.answer}
+    question: ${inputs.question}
+    context: ${inputs.context}
+    temperature: 0
+  connection: open_ai_connection
+  api: chat

examples/flows/evaluation/eval-perceived-intelligence/gpt_perceived_intelligence.md

@@ -0,0 +1,23 @@
+# user:
+# Instructions
+
+* There are many chatbots that can answer users questions based on the context given from different sources like search results, or snippets from books/papers. They try to understand users's question and then get context by either performing search from search engines, databases or books/papers for relevant content. Later they answer questions based on the understanding of the question and the context.
+* Perceived intelligence is the degree to which a bot can impress the user with its responses, by showing originality, insight, creativity, knowledge, and adaptability. Perceived intelligence can be influenced by various factors, such as the content, tone, style, and structure of the bot's responses, the relevance, coherence, and accuracy of the information the bot provides, the creativity, originality, and wit of the bot's expressions, the depth, breadth, and insight of the bot's knowledge, and the ability of the bot to adapt, learn, and use feedback.
+* Your goal is to score the answer for given question and context from 1 to 10 based on perceived intelligence described above:
+    * Score 10 means the answer is excellent for perceived intelligence
+    * Score 1 means the answer is poor for perceived intelligence
+    * Score 5 means the answer is normal for perceived intelligence
+* Just respond with the score, nothing else.
+
+# Real work
+
+## Question
+{{question}}
+
+## Answer
+{{answer}}
+
+## Context
+{{context}}
+
+## Score

examples/flows/evaluation/eval-perceived-intelligence/parse_score.py

@@ -0,0 +1,15 @@
+from promptflow.core import tool
+import re
+
+
+@tool
+def parse_score(gpt_score: str):
+    return float(extract_float(gpt_score))
+
+
+def extract_float(s):
+    match = re.search(r"[-+]?\d*\.\d+|\d+", s)
+    if match:
+        return float(match.group())
+    else:
+        return None

examples/flows/evaluation/eval-perceived-intelligence/requirements.txt

@@ -0,0 +1,2 @@
+promptflow
+promptflow-tools